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Abstract 


This  report  is  a  delineation  of  research  results  derived  under  the 
sponsorship  of  the  AFOSR  grant  during  January  1984  through  January 
1987.  For  the  sake  of  conciseness,  results  already  published  in  open 
literature  are  only  abstracted.  Other  results  that  are  yet  to  appear  are 
described  in  more  detail. 
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1  Introduction 


This  final  report  for  AFOSR  84-0052  provides  a  summary  of  various  research  activities 
carried  out  during  three  years  of  sponsorship. 

The  research  proposed  sought  to  focus  on  the  following: 

1.1 

Study  and  development  of  certain  fault-tolerant  architectures  that  utilize  the  capabil¬ 
ities  of  the  new  IC  technology  was  undertaken.  Specifically,  the  research  was  aimed 
at  network  architectures,  distinguished  by  a  close  interconnection  of  a  large  number  of 
computing  elements.  Included  is  a  subclass  of  specialized  network  architectures  known 
as  VLSI  processor  arrays.  Besides  fault-tolerance-related  research  for  such  arrays, 
also  proposed  was  exploration  of  a  new  array  architecture,  developed  for  the  express 
purpose  of  executing  general  algorithms  on  these  arrays. 

The  precise  research  formulated  -  developing  fault-tolerant  multiprocessor  net¬ 
work  architectures  -  goes  beyond  earlier  work.  Here,  the  system  interconnection  struc¬ 
ture,  itself,  was  used  as  the  primary  design  tool  for  achieving  various  and  diverse 
objectives,  including:  low  interconnection  and  layout  complexities,  dynamic  reconfig¬ 
urability,  fault-tolerance  through  graceful  degradation  as  well  as  self-diagnosability. 
Viability  of  the  proposed  research  was  demonstated  in  the  proposal;  new  communi¬ 
cation  structures  were  introduced,  along  with  concepts  of  admissability  of  multiple 
logical  configurations,  and  algorithmic  and  detour  routing  that  provide  fault-tolerance 

1 


and  graceful  degradation. 


1.2 

It  was  then  proposed  that  the  research  be  extended  to  the  study  of  the  design  of 
certain  VLSI  processor  arrays,  particularly  because  this  subject  matter  was  so  new,  and 
only  very  limited  work  had  yet  been  reported.  Fundamental  concepts  were  proposed 
identifying  important  relationships  between  various  levels  of  redundancy  and  fault- 
tolerance.  Also  achieving  testability  and  diagnosability  internally,  within  the  arrays, 
was  sought. 

1.3 


A  marked  limitation  of  earlier  VLSI  arrays  had  been  suitability  only  for  very  spe¬ 
cialized  and  highly  concurrent  matrix-oriented  computation.  An  entirely  new  way  of 
utilizing  these  arrays  was  what  was  proposed  here,  which  allow  for  execution  of  general 
algorithms,  thereby  making  these  arrays  attractive  for  broader  use. 

This  report  is  organized  into  three  main  sections.  Section  2  highlights  various 
results  obtained  on  these  above  topics.  Section  3  provides  a  complete  list  of  publications 
and  student  support  resulting  from  this  research.  Section  4  depicts  several  unresolved 


future  research  issues. 


2  Review  of  Research  Results 


Below  is  delineated  a  summary  of  various  research  results  derived  under  the  grant. 
Those  already  published  in  open  literature  are  only  abstracted  here;  those  yet  to  appear 
are  described  in  some  detail. 

Fault- Tolerant  Multiprocessor  Networks 

In  [Prad85a|,  a  class  of  link-  and  bus-oriented  regular  networks  was  presented. 
Significantly,  these  were  shown  to  provide  optimal/near  optimal  fault-tolerance.  Vari¬ 
ous  fault-tolerant  properties  of  these  networks  were  analyzed  extensively,  as  well,  com¬ 
pared  against  existing  networks.  What  is  novel  is  the  capacity  of  these  networks  to 
be  used  to  design  any  arbitrarily  large  networks,  by  using  building  blocks  of  any  given 
number  of  connections  per  node.  (Other  fault-tolerant  networks,  like  binary  cube, 
suffer  from  the  so-called  “fan-out”  problem,  requiring  that  the  number  of  connections 
per  node  be  increased  with  the  size  of  the  network). 

In  a  subsequent  paper,  [Prad85b],  a  dynamically  restructurable  fault-tolerant 
processor  network  architecture  was  presented.  What  is  significant  about  these  type  of 
networks  is  that  the  inherent  logical  structure  can  be  changed  to  fit  the  application  in 
hand.  Consequently,  the  proposed  network  admits  efficient  execution  of  a  large  class  of 
algorithms.  Even  more  importantly,  these  networks  admit  a  measure  of  fault-tolerance 
because  the  faulty  network  exhibits  all  of  the  important  properties  of  the  fault-free 
network. 

Of  special  note  is  a  recommendation  that  this  particular  network  (Prad85b)  has 


significant  potential  for  SDI  application,  from  a  recent  study  conducted  by  Control 
Data  under  the  sponsorship  of  Rome  Air  Development  Center  (Appendix). 

Several  bus  oriented  fault-tolerant  networks  were  reported  on,  as  well,  in  (PrSc84). 

Key  considerations  in  the  design  of  fault-tolerant  multiprocessor  systems  are 
testing  and  diagnosis.  A  fault-tolerant  system  must  also  be  testable  with  a  high  degree 
of  confidence.  Reliability  of  the  system,  itself,  otherwise  is  compromised.  Distributed 
self-diagnosis  is  a  promising  approach  to  testing/diagnosis  problems.  Here,  by  inter¬ 
rupting  the  computation,  the  processors  are  able  to  test  certain  of  the  other  processors, 
determining  if  they  are  faulty.  Interesting  in  this  approach  is  its  ability  to  be  used 
in  both  acceptance  testing  and  concurrent  testing  of  multiprocessor  systems.  A  new 
methodology  was  pursued,  with  the  objective  of  minimizing  testing  overhead  as  well  as 
of  achieving  greater  test  reliability,  and/or  more  frequent  testing.  Several  important 
results  were  published  in  [MePr85]. 

DeBruijn  Multiprocessor  Networks 

Certain  interesting  work  on  networks  able  to  solve  a  wide  variety  of  problems 
evolved,  described  in  some  detail  below. 

Successfully  grouped  into  various  classes  are  computational  problems.  These  are 
important  classifications,  include  the  pipeline  class ,  multiplex  class,  the  NP-complete 
class,  the  ASCEND  and  DESCEND  classes,  as  well  as  the  decomposable  searching  class. 

Problems  in  the  pipeline  class  can  be  efficiently  solved  in  a  pipe  (known  also  as 
a  linear  array).  Depending  on  the  problem,  data  may  flow  in  one  direction  or  in  both 


directions,  simultaneously.  Matrix-vector  multiplication  is  a  typical  example  of  prob¬ 
lems  solvable  with  one-way  pipeline  algorithms.  Band  matrix-vector  multiplication, 
recurrence  evaluation  and  priority  queues  are  problems  that  can  be  solved  by  two-way 
pipeline  algorithms. 

The  multiplex  class  covers  a  range  of  problems  characterized  by:  (1)  Operation 
on  N  data  operands  to  produce  a  single  result;  (2)  Evaluation  can  be  described  by  a 
tree.  This  category  includes  evaluation  of  general  arithmetic  expressions,  polynomial 
evaluation,  etc.  The  natural  computation  graph  for  this  paradigm  is  a  tree,  whose 
nodes  correspond  to  operations,  and  whose  edges  correspond  to  dataflow  between  op¬ 
erations.  The  CBT  (complete  binary  tree)  can  be  used  to  solve  the  problems  belonging 
to  this  class.  For  another  important  class  of  problems,  the  NP-complete  class ,  the  CBT 
can  efficiently  implement  exhaustive  search  algorithms,  where  time  complexity  still  is 
exponential. 

The  ASCEND  and  DESCEND  classes  are  comprised  of  highly  parallel  algo¬ 
rithms.  The  paradigm  of  the  algorithms  in  this  class  is  the  iterative  rendition  of  a 
divide-and-conquer  scheme.  The  input  and  output  are  each  a  vector  of  N(=  2k)  data 
items;  “divide”  refers  to  two  subproblems  of  equal  size;  the  “marry  step”  combines  the 
results  of  two  subproblems,  executing  a  single  operation  on  corresponding  pairs  of  data 
items.  Assume  that  input  data  D0,  are  stored,  respectively,  in  storage  loca¬ 

tions  T[0],  T[l], .  .  .  ,T[N-1).  An  algorithm  in  the  DESCEND  class  performs  a  sequence 
of  basic  operations  on  pairs  of  data  that  are  successively  2K~l,2k~l,  ...,2*, 2°  locations 
apart.  In  terms  of  the  above  divide-and-conquer  model,  the  marry  step  involves  pairs 


of  2°  locations  apart.  In  the  dual  class  (the  ASCEND  class),  the  basic  operations  are 
performed  on  the  data  that  are  successively  2°,  21, 2*-1  locations  apart;  the  marry 
step  involves  pairs  of  2*_1  locations  apart.  These  problems  can  be  solved  in  the  SE  and 
the  CCC. 


Problems  in  the  decomposable  starching  class  can  be  described  as  illustrated. 
Preprocess  a  set  F  of  N  objects  into  a  data  structure  D,  such  that  certain  kinds  of 
queries  about  F  can  be  answered  quickly.  A  searching  problem  is  decomposable  if  the 
response  to  a  query  Q,  asking  the  relation  of  an  object  z  to  the  set  F,  can  be  written 
as:  Q(z ,  F)  —  6q(z,  /),  for  all  f  in  F,  where  f  is  an  element  in  F,  6  is  a  binary  operator 
which  is  associative,  commutative  and  has  an  identity,  and  where  q  is  the  query  asking 
the  relation  of  the  object  z  to  the  element  f.  The  TM  described  solves  this  large  class 
of  searching  problems. 

Multiprocessor  networks  based  on  undirected  binary  de  Bruijn  graphs,  able  to 
solve  all  of  the  above  mentioned  classes  of  problems  were  presented  in  [PrSa87].  No 
other  such  network  has,  we  believe,  yet  been  identified.  Another  paper  [PrSa85]  pre¬ 
sented  a  corollary  of  the  above  work,  demonstrating  that  these  networks  can  perform 
efficient  sorting  algorithms  in  various  different  input/output  modes. 

Also  last  year,  in  collaboration  with  researchers  from  the  University  of  Wiscon¬ 
sin,  we  studied  the  problem  of  reconfiguration  of  interleaved  memory  in  the  presence  of 
faults.  These  results  will  appear  shortly  in  the  1987  IEEE/ACM  Computer  Architec¬ 
ture  Symposium,  and  are  described  below  in  some  detail,  not  reported  on  previously. 

In  a  computer  system  that  consists  of  a  processing  unit  (CPU)  connected  to  a 
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memory  system,  the  rate  at  which  the  CPU  can  process  information  is  limited  by  the 
rate  at  which  the  memory  can  supply  this  information.  Furthermore,  the  information¬ 
processing  rate  of  the  CPU  is  also  limited  by  the  bandwidth  of  the  interface  between  the 
CPU  and  the  memory.  This  is  the  well-known  von  Neumann  bottleneck.  Consequently, 
a  decrease  in  the  bandwidth  of  a  memory  system  will  directly  affect  the  performance 
of  the  overall  computer  system. 

There  are  two  main  approaches  to  attain  a  memory  system  with  a  high  band¬ 
width.  The  first  involves  the  use  of  a  high-speed  buffer  or  cache  memory  and  the  second 
involves  the  use  of  several  memory  banks  connected  in  an  interleaved  fashion.  Though 
the  use  of  cache  memories  has  become  widespread,  their  utility  is  limited  by  their  size. 
While  cache  memories  are  very  effective  for  instructions  and  scalar  data  items,  they 
have  not  proven  to  be  effective  for  numeric  processing  machines  that  utilize  large  data 
structures  (such  as  arrays).  For  such  systems,  in  order  to  achieve  a  high-bandwidth 
memory  system,  one  is  forced  to  use  interleaved  banks  of  memory.  Of  course,  the  best 
effect  is  achieved  by  using  a  cache  memory  for  instruction  buffers  (analogous  to  an 
instruction  cache),  large  B  and  T  register  file  (analogous  to  a  cache  for  scalar  data) 
and  an  interleaved  memory  for  non-scalar  data. 

In  an  interleaved  memory  system  that  consists  of  N  independent  memory  banks 
(or  modules),  by  associating  address  latches  and  data  latches  with  each  bank,  N  differ¬ 
ent  memory  accesses  can  be  carried  out  simultaneously.  By  doing  so,  the  bandwidth 
of  the  memory  system  can  be  increased  to  N  times  the  bandwidth  of  a  single  bank. 
A  processing  system  that  utilizes  a  cache  memory  for  instructions  and  an  interleaved 


Figure  1:  A  Processor  with  an  Interleaved  Memory  System 


memory  system  for  data  is  shown  in  Figure  1.  The  bandwidth  of  interleaved  memo¬ 
ries  has  been  the  subject  of  extensive  study.  Apart  from  the  referencing  behavior  of 
programs,  the  main  factor  that  influences  the  bandwidth  of  interleaved  memory  banks 
is  the  manner  in  which  the  addresses  are  distributed  amongst  the  banks.  Given  the 
distribution  of  data  amongst  the  memory  banks,  the  appropriate  address  bits  can  be 
used  to  select  the  bank  that  contains  the  desired  data  item.  Generally  the  number  of 
banks,  N  that  are  used  to  build  an  interleaved  memory  is  a  power  of  2,  i.e.,  N  =  2* 
where  q  is  an  integer.  In  such  a  system,  q  bits  of  the  address  suffice  to  select  a  bank 
and  the  remaining  bits  are  used  to  select  a  word  within  a  bank.  If  the  q  bits  are  the 
high-order  bits  of  the  address  space,  the  scheme  is  a  high-order  interleaving  scheme 
whereas  a  low-order  interleaving  scheme  results  if  the  low-order  q  bits  are  used  to  select 


We  should  mention  that  an  interleaving  scheme  is  not  restricted  to  using  only  a 
power  of  2  number  banks.  Interleaving  schemes  that  utilize  a  prime  number  of  memory 
banks  have  been  investigated  and  implemented.  However,  the  utility  of  such  a  scheme 
in  high  performance  machines  is  limited  because  of  the  complex  logic  that  is  needed  to 
determine  the  appropriate  bank/ word  from  a  given  address. 

In  a  high-order  interleaved  memory  system,  consecutive  memory  adddresses  lie 
in  the  same  bank.  Therefore,  if  the  memory  is  referenced  sequentially,  consecutive 
memory  references  access  the  same  bank  and  no  increase  in  bandwidth  is  obtained.  In 
a  low-order  interleaved  memory  system,  consecutive  addresses  lie  in  different  banks. 
Now,  if  the  memory  is  accessed  sequentially,  consecutive  references  will  access  different 
banks,  thereby  increasing  the  bandwidth  of  the  memory.  Since  the  memory  referencing 
pattern  for  most  programs  is  generally  sequential  (because  of  sequential  instructions 
and  array  structures  with  a  constant  stride  of  unity),  a  low-order  interleaved  memory 
system  generally  has  a  higher  bandwidth  than  a  high-order  interleaved  memory  system. 

A  low-order  interleaving  scheme  has  a  major  drawback  -  it  is  not  modular, 
i.e.,  a  failure  in  a  single  bank  affects  the  entire  address  space.  If  no  precautions  are 
taken  to  handle  such  a  situation,  the  bandwidth  of  the  memory  and  consequently  the 
performance  of  the  processor  could  be  degraded  to  an  intolerable  extent.  In  this  paper, 
we  study  the  organization  of  interleaved  memories  such  that  faults  in  the  memory 
system  degrade  the  performance  in  a  graceful  manner.  We  restrict  our  study  to  an 
interleaved  memory  system  that  starts  out  with  a  power  of  2  number  of  banks  and  uses 
a  low-order  interleaving  scheme.  The  ideas  presented  in  (Pr  et.al.87]  can  be  extended 


to  other  interleaved  memory  schemes. 


Faults  in  Interleaved  Memories 

Consider  a  memory  that  consists  of  several  groups  of  interleaved  memories  with 
each  group  consisting  of  several  banks.  The  number  of  banks  in  a  group  is  a  power  of 
2,  say  2r,  and  the  banks  within  a  group  are  fully  interleaved.  Thus,  the  banks  within  a 
group  can  be  selected  using  a  r-bit  bank  selection  address  held.  Different  groups  can 
have  a  different  number  of  banks  in  them.  Thus  group  G j  may  consist  of  4  banks  while 
group  may  have  only  1  bank.  If  the  total  number  of  banks  in  the  memory  system  is 
2*  where  A:  is  an  integer,  then  there  is  only  one  group.  This  is  the  situation  that  exists  in 
a  conventional  interleaved  memory  system  without  any  faulty  banks.  Therefore,  if  each 
of  the  2*  banks  in  the  single  group  contains  2P  words,  then  the  total  addressable  main 
memory  of  the  system  is  2g  (where  n  =  p  +  q)  words.  Using  a  low-  order  interleaving 
scheme,  bits  A,_!  .  .  .A0  of  the  n-bit  address  An_i  An_2  •  -  •  A0  (where  An-i  is  the 
most  significant  bit)  are  used  to  select  the  bank  and  the  remaining  p  bits,  i.e.,  bits 
A„_i  2  .  .  .A,  are  used  to  select  a  word  within  a  bank. 

Consider  what  happens  when  a  bank  is  deleted  from  a  memory  system  that 
contains  a  single  group  of  banks.  This  is  exactly  what  happens  when  a  fault  in  the 
memory  system  results  into  the  loss  of  one  complete  bank.  Therefore,  our  fault  model 
is  that  a  fault  results  in  the  loss  of  a  complete  bank  of  memory.  We  assume  that  a 
mechanism  that  detects  the  presence  of  a  faulty  bank  exists.  Such  a  fault-detection 
scheme  is  not  the  subject  matter  of  this  paper.,  Our  main  thrust  is  to  evaluate  the 
loss  in  performance  when  a  fault  is  reported  and  how  the  memory  system  might  be 
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organized  so  that  the  resulting  degradation  in  performance  is  graceful. 

If  the  memory  system  loses  one  bank,  the  number  of  banks  in  memory  is  reduced 
to  2*  -  1  and  the  total  addressable  physical  memory  is  reduced  to  (2*  -  1)2P  words.  The 
program  must  be  stopped,  correct  information  recovered  from  the  backup  store,  the 
address  translation  mechanism  informed  about  the  faulty  bank  and  program  execution 
restored.  However,  since  V  —  1  is  not  a  power  of  2,  the  banks  no  longer  form  a 
single  group  and  the  system  loses  its  capacity  to  interleave  memory  requests.  Without 
interleaving,  the  bandwidth  can  be  catastrophic  to  the  performance  of  a  high-speed 
CPU.  What  could  we  possibly  do  to  salvage  some  of  this  lost  memory  performance? 
Two  approaches  follow. 

The  first  approach  involves  the  use  of  spare  memory  banks.  After  a  faulty  bank 
is  detected,  a  spare  bank  can  take  its  place.  However,  as  more  banks  become  faulty,  the 
system  will  eventually  run  out  of  spare  banks  if  the  spare  banks  cannot  be  replaced. 
Once  all  the  spare  banks  have  been  exhausted,  another  fault-tolerance  scheme  must 
come  into  play. 

Reconfiguration  of  Non-faulty  Banks 

An  alternative  approach  is  to  reconfigure  the  remaining  non-faulty  banks  in 
order  to  salvage  some  of  the  lost  performance.  Such  an  approach  could  also  be  used  if 
a  system  has  spares  but  runs  out  of  them  eventually.  The  banks  are  reconfigured  so 
that  their  bandwidth  is  improved.  No  doubt,  a  smaller  physical  memory  will  result  in 
a  larger  probability  of  a  page  fault. 

How  might  we  organize  the  fault-free  banks  so  that  the  performance  is  not 


degraded  to  an  intolerable  extent?  A  simple  solution  that  could  be  used  to  salvage 
some  of  the  lost  bandwidth  is  to  reduce  the  number  of  addressable  banks  to  the  nearest 
power  of  two,  i.e.,  2q~1  thereby  achieving  a  maximum  bandwidth  of  2,_1  words  per 
memory  cycle.  While  the  address  translation  and  bank  selection  mechanism  is  quite 
straight-forward,  2q~l  -  1  banks  of  fault-free  physical  memory  are  not  addressable 
and  therefore  unutilized.  With  such  a  simple  reconfiguration  scheme,  although  the 
bandwidth  may  be  high,  it  is  likely  to  result  into  a  high  page  fault  rate  compared  to  a 
scheme  which  uses  all  2q  —  1  fault  free  memory  banks.  A  scheme  that  does  not  utilize 
all  2q  —  1  memory  banks  may,  therefore,  perform  poorly  in  spite  of  its  high  bandwidth. 
Hence,  any  scheme  used  to  improve  the  memory  performance  must  not  only  organize 
the  fault-free  banks  in  an  interleaved  manner  to  maximize  the  bandwidth,  but  it  must 
also  make  sure  that  the  available  memory  is  being  utilized  to  its  fullest  extent  so  that 
the  performance  degradation  due  to  page  faults  is  minimized. 

Another  important  factor  that  must  be  kept  in  mind  is  the  effect  of  the  reconfig¬ 
uration  hardware  on  delay  in  the  address  and  data  path.  One  advantage  of  a  low-order 
interleaving  scheme  is  that  the  decoding  logic  needed  to  generate  the  bank  select  sig¬ 
nals  is  very  simple.  Therefore,  the  delay  in  the  address  path  between  the  processor  and 
main  memory  is  small.  It  is  desirable  to  keep  this  delay  small  because  any  delay  in  this 
path  will  have  a  direct  impact  on  the  latency  of  each  memory  request.  In  the  proposed 
reconfiguration,  we  pay  special  attention  to  this  critical  path  performance.  Thus,  in 
addition  to  making  the  best  use  of  available  memory  resources,  we  must  also  minimize 


the  delay  due  to  the  additional  hardware. 


Figure  2:  Partitioning  Banks  into  Groups 
The  Reconfiguration  Scheme 

The  proposed  scheme  reconfigures  the  remaining  banks  using  a  combination  of 
high-order  interleaving  and  low-order  interleaving.  All  non-faulty  banks  are  parti¬ 
tioned  into  sets.  Thus,  if  2q~x  -  1  banks  were  available,  they  would  be  partitioned  into 
q  sets.  These  q  sets  form  2  subsets;  subset  S0{2q~2)  and  2,_2  —  1).  A  set  containing 
a  power  of  2  banks  is  called  a  group.  Therefore,  50(2,_1)  has  one  group  G0(2,_l)  that 
has  2,_1  banks  and  5j(2,_1  —  1)  is  made  up  of  group  Gio(2,_2)  which  has  2q~2  banks 
and  the  subset  Si(2,_2  -  1)  which  has  ( 2q~l  -  1)  banks.  This  recursive  partition  stops 
when  Si  has  only  one  bank.  Clearly,  the  number  of  banks  in  each  group  is  a  power 
of  2,  with  unity  being  a  special  case.  An  example  of  the  partitioning  of  7  memory 
banks  into  groups  is  given  in  Figure  2.  Banks  within  each  group  G,( 2*)  are  organized 
for  low-order  interleaving;  high-order  address  bits  are  used  to  determmine  the  group. 
If  there  is  only  one  group,  e.g.,  in  the  fault-free  case,  no  group  selection  needs  to  be 
done.  The  low-order  q  bits  of  the  address  select  the  bank  and  the  high-order  p  bits 


of  the  address  select  the  word  within  the  bank.  With  one  fault,  the  number  of  groups 
becomes  q  with  the  number  of  banks  =  2q  —  1.  Therefore,  q  bits  suffice  to  uniquely 
identify  2q  —  1  correct  and  one  faulty  bank.  An  address  is  decoded  as  follows:  the  most 
significant  bit  of  the  address,  An- 1,  is  used  to  select  either  group  G0(2,_1)  or  subset 
51(2«~1  -  l).  If  group  Go(2,_1)  ‘s  selected,  then  bits  Aq-2...Ao  are  used  to  select  one 
of  the  2q~1  banks  within  the  group  and  bits  An_2...A,_i  are  used  to  address  the  word 
within  the  bank.  If  <?i(2?-1  —  1)  is  selected,  then  bit  An_2  of  the  address  is  used  to 
select  either  Gio(2,“l)  (with  bits  Aq-a...Ao  used  to  select  a  bank  within  this  group)  or 
5j(2,-J)  and  so  on.  Note  that  this  group  identification  scheme  resembles  the  decoding 
scheme  used  to  decode  Huffman-encoded  information.  If  there  is  only  a  single  faulty 
bank,  it  is  always  indicated  by  a  string  of  l’s  in  the  q  bits  that  are  used  to  identify 
the  bank  number.  Once  the  group  number  has  been  determined  from  the  address,  the 
appropriate  p  bits  are  used  to  select  the  work  within  the  bank.  The  logic  that  decodes 
the  address  is  now  more  complex  than  a  simple  decoder.  We  call  this  logic  the  Ad¬ 
dress  Transliterator  (AT).  Each  memory  address  now  passes  through  the  AT  before  it 
is  forwarded  to  the  memory  system  (Figure  3).  The  design  of  the  AT  is  discussed  in 
detail  in  Section  4.  The  inputs  to  the  AT  are  n-bit  physical  memory  address  and  a 
2,-bit  vector,  the  Bank  Status  Indicator  (BSI),  that  indicates  the  status  of  each  bank. 
The  output  from  the  AT  is  the  appropriate  bank  address  and  the  address  of  the  work 
within  the  bank. 

Performance  Evaluation 

In  order  to  evaluate  the  performance  of  the  reconfigured  memory  system,  we 
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Figure  3:  Interleaved  Memory  System  with  an  Address  Transliterator 

carried  out  a  trace-driven  simulation  of  several  programs  on  the  VAX-11.  A  trace 
of  instruction  and  data  references  were  obtained  for  each  program.  Data  from  the 
trace  files  was  fed  into  a  program  that  simulates  the  interleaved  memory  system  and 
determines  the  bandwidth.  The  simulation  model  used  was  similar  to  that  used  by 
other  researchers.  Memory  references  are  divided  into  instruction  references  (put  in  the 
instruction  queue)  and  data  references  (put  in  the  data  queue).  During  each  memory 
cycle,  requests  in  both  queues  are  scanned  alternately.  The  scanner  stops  admitting 
requests  if  a  bank  conflict  occurs.  We  make  the  following  assumptions;  no  interactions 
may  occur  between  the  instruction  request  stream  and  the  data  request  stream,  ii)  no 
self-modifying  code  and  iii)  an  additional  data  cycle  is  issued  if  the  data  queue  is  full. 

The  Performance  Metric 

Along  with  the  bandwidth  of  the  reconfigured  memory  system,  the  other  im- 


portant  performance  measure  is  the  number  of  page  faults.  We  combine  these  two 
measures  into  a  single  metric,  P.  The  performance  metric  P  is  defined  as: 

•  P=total  data  trace  length  (total  data  references/data  bandwidth)  x  data  pages 
allocated  to  a  process  -f-  time  to  process  a  page  fault  x  data  pages  allocated  to  a 
process  x  number  of  data  page  faults. 

The  metric  P  is  then  normalized  with  respect  to  P  for  the  fault-free  case. 
Experiments  and  Results 

We  evaluated  the  reconfigured  memory  system  for  4  different  programs:  i)  nroff, 
which  is  a  text  formatter,  ii)  compact,  which  is  a  program  that  compresses  a  file  using 
an  adaptive  Huffman  encoding,  iii)  boyer,  which  is  a  theorem  proving  program  and,  iv) 
tak,  which  is  an  execution  of  the  Takeuchi  function.  Initially,  16  banks  of  memory  are 
present.  The  system  uses  demand  paging  with  a  page  size  of  2K  bytes  and  a  bus  width 
of  4  bytes.  The  number  of  instruction  and  data  references  traced  and  the  number  of 
data  pages  used  during  a  trace  of  the  program  is  given  in  Table  1. 

A  program  is  allocated  a  fixed  number  of  data  pages  (maximum  of  32)  for 
its  use.  A  least  recently  used  (LRU)  replacement  policy  is  used  to  replace  a  page 
when  a  page  fault  occurs  and  no  free  page  frame  is  available.  We  assume  that  all 
instruction  references  are  serviced  by  the  instruction  cache,  i.e.,  only  data  requests  to 
the  interleaved  memory.  A  loss  of  memory  bank  results  in  less  data  pages  available 
for  the  program,  but  does  not  affect  the  instruction  pages.  Thus,  the  number  of  data 
pages  allocated  to  a  program  is  reduced  when  a  bank  is  reported  to  be  faulty.  The 


Trace 

Trace  Records 
Instruction  Data 

Data  Pages 
Referenced 

rroff 

compact 

boyer 

tak 

281513  178832 
233638  205298 
217147  229871 
49814  54590 

MM 

Table  1:  Statistics  for  the  Benchmark  Programs 

pages  are  distributed  amongst  the  groups  of  the  reconfigured  memory  in  proportion 
to  the  number  of  banks  in  the  group.  Any  page  lies  completely  within  a  group.  For 
example,  if  there  are  2  groups  consisting  of  8  and  4  banks  respectively,  a  process  will 
place  67%  of  its  data  pages  in  the  group  of  8  banks  and  the  remaining  pages  in  the 
group  of  4  banks.  The  time  to  process  a  page  fault  is  2000  memory  cycles. 

Using  the  above  parameters,  we  calculated  the  value  of  P  as  the  number  of 
addressable  memory  banks  reduced.  The  results  are  presented  in  Tables  2-5.  In  all 
cases,  there  is  a  significant  increase  in  the  page  fault  rate  when  fewer  memory  banks 
(pages)  are  available.  The  decrease  in  data  bandwidth,  however,  is  not  very  significant. 
The  performance  metric  P  degrades  in  a  graceful  manner  as  opposed  to  a  sudden 
change  if  the  number  of  addressable  memory  banks  was  reduced  to  8  when  a  single 
bank  became  faulty. 


Number 
of  Banks 

Bandwidth 

Page  Faults 

Data 

Page  Fault  rate  (%) 

Pages 

P 

(normalized) 

16 

1934366 

58 

0.032433 

28 

1.000000 

15 

2.744169 

83 

0.046412 

26 

1.213130 

14 

2.738286 

136 

0.076049 

24 

1.633969 

13 

2.740889 

141 

0.078845 

22 

1.541935 

12 

2.735506 

173 

0.096739 

20 

1.660630 

11 

2.663697 

217 

0.121343 

18 

1.820683 

10 

2.698165 

300 

0.167755 

16 

2.151699 

9 

1675584 

362 

0.202425 

14 

2.234710 

8 

1614926 

1180 

0.659837 

12 

5.881731 

Table  2:  Result  for  nroff 


Number 
of  Banks 

Bandwidth 

Page  Faults 

Data 

Page  Fault  rate  (%) 

Pages 

P 

(normalized) 

16 

2.906916 

23 

0.011203 

14 

1.000000 

15 

1.945492 

51 

0.024842 

13 

1.652334 

14 

1.959661 

171 

0.083294 

12 

3.283534 

13 

2.796937 

467 

0.227474 

11 

6.787020 

12 

2.698449 

1048 

0.510477 

10 

13.303313 

11 

2.585389 

2334 

1.136884 

9 

26.168747 

10 

2.648802 

3986 

1.941567 

8 

39.440582 

9 

2.573625 

5382 

2.621555 

7 

46.490295 

8 

2.546427 

6497 

3.164668 

6 

48.046799 

Table  3:  Result  for  compact 


Number 
of  Banks 

Bandwidth 

Page  Faults 

Data 

Page  Fault  rate  (%) 

Pages 

P 

(normalized) 

16 

3.283168 

1465 

0.637314 

32 

1.000000 

15 

2.808648 

16% 

0.737805 

30 

1.085571 

14 

3.109895 

1%2 

0.853522 

28 

1.166053 

13 

3.042513 

2328 

1.012742 

26 

1.281456 

12 

3.142504 

2754 

1.198063 

24 

1.395280 

11 

2.957110 

3209 

1.396000 

22 

1.488599 

10 

2.997978 

4454 

1.937609 

20 

1.871798 

9 

2.223019 

66% 

2.912938 

18 

2.530375 

8 

2.997796 

8895 

3.869561 

16 

2.977763 

Table  4:  Result  for  boycr 
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Number 

Data 

P 

of  Banks 

Bandwidth 

Page  Faults 

Pace  Fault  rate  (%) 

Pages 

■tESnEi 

16 

3.687766 

605 

1.108262 

32 

1.000000 

15 

2.936841 

674 

1.234658 

30 

1.046026 

14 

3.110719 

759 

1.390365 

28 

1.096997 

13 

2.535061 

823 

1.507602 

•26 

1.106195 

12 

3.379349 

895 

1.639494 

24 

1.105987 

11 

3.261051 

993 

1.819015 

22 

1.124167 

10 

2.967439 

1112 

2.037004 

20 

1.144263 

9 

3.130943 

1198 

2.194541 

18 

1.108388 

8 

3.266726 

1389 

2.544422 

16 

1.140882 
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4  Research  in  Progress  and  Future  Directions 


4.1  Consensus  with  Dual  Failure  Modes 

Consider  a  distributed  system  of  processing  elements  (nodes)  connected  by  a  point- 
to-point  network.  One  of  the  common  problems  is  to  maintain  clocks  (or  some  other 
concept  of  time)  in  approximate  synchrony  despite  differences  in  the  clock  rates  of 
individual  processors  and  despite  some  faulty  processors. 

We  consider  a  generalization  of  the  clock  synchronization  problem  called  inter¬ 
active  consistency  that  involves  reaching  agreement  on  some  value  being  sent  by  one  of 
the  processors,  say  s.  Let  i/(s)  be  the  value  that  s  wishes  to  transmit.  Each  processor 
decides  on  some  value  as  having  been  sent  by  s. 

Interactive  consistency  is  defined  as  satisfying  the  following  two  requirements: 

•  agreement  any  two  good  processors  decide  that  the  same  value  was  sent  by  s 

•  sanity  if  s  is  a  good  processor,  then  the  value  that  any  good  processor  decides 
was  sent  by  s  is,  in  fact,  v(s) 

The  agreement  requirement  ensures  that  faulty  processors  cannot  cause  two 
good  processors  to  “believe”  different  things.  The  sanity  requirement  ensures  that 
consistency  cannot  be  achieved  by  simply  agreeing  on  a  default  value. 

Figure  4  shows  why  it  is  impossible  to  reach  agreement  in  the  presence  of  one 
(malicious)  fault  when  there  are  only  three  processors.  The  left  illustration  depicts  the 
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nature  of  messages  sent  when  node  t  is  transmitting  the  value  1  and  node  j  is  faulty. 
The  right  illustration  depicts  the  nature  of  messages  sent  when  node  t  is  faulty  and 
sends  conflicting  values  to  j  and  k.  Node  k  is  unable  to  distinguish  between  these  two 
scenarios. 


Figure  4:  Agreement  impossible  with  only  three  nodes 


We  consider  systems  where  either  or  both  of  two  types  of  faults  may  occur: 
benign  or  malicious.  We  have  accomplished  the  following:  (1)  established  a  taxonomy 
of  processor  failure  modes,  (2)  established  a  continuum  between  previous  results  on 
necessary  and  sufficient  conditions  for  consensus  algorithms  to  exist,  and  (3)  developed 
more  general  algorithms  to  handle  systems  with  two  failure  modes — achieving  much 
improved  reliability. 

We  have  established  a  taxonomy  that  categorizes  faults  by  their  behavior.  (See 
F.  Meyer  and  D.  Pradhan,  “Consensus  with  dual  failure  modes,”  Proc.  17th  Fault- 
Tolerant  Comput.  Stjmp.,  for  details).  There  are  many  possible  failure  modes.  The 
three  most  thoroughly  examined  failure  modes  are:  (1)  Byzantine,  (2)  Authenticated, 


and  (3)  Dormant.  Figures  5,  6,  and  7  illustrate  the  sort  of  behavior  that  such  faults 
may  exhibit.  The  Dormant  (Byzantine)  failure  mode  has  the  most  (least)  constraints 
on  its  behavior. 


Figure  5:  Byzantine  fault  exhibits  arbitrary  behavior 

We  have  taken  malicious  failures  to  be  Byzantine  and  benign  failures  to  be 
Dormant.  Previous  researchers  have  developed  algorithms  that  function  in  the  presence 
of  one  of  these  two  failure  modes.  F.  Cristian,  et  al,  “Atomic  broadcast:  From  simple 
message  diffusion  to  Byzantine  agreement,”  Proc.  15th  Fault- Tolerant  Comput.  Symp. 
gave  a  very  simple  algorithm  that  contends  with  any  number  of  benign  failures.  But  the 
algorithm  almost  always  fails  whenever  a  failed  processor  exhibits  any  behavior  outside 
the  constraints  of  a  dormant  failure  (for  instance,  if  it  sends  an  incorrect  message).  D. 
Dolev,  et  al,  “An  efficient  algorithm  for  Byzantine  agreement  without  authentication,” 
Information  and  Control ,  no.  52,  gave  a  relatively  efficient  algorithm  to  contend  with 


Figure  6:  Authenticated  fault  is  constrained  by  encrypted  messages 


v(s)  =  1 


omitted  message 


Figure  7:  Dormant  fault  commits  only  faults  of  omission 


a  maximal  number  (about  one-third  of  the  processors)  of  malicious  failures. 


Figure  8  plots  the  reliabilities  achieved  by  these  two  algorithms.  The  plot  is  for 
a  64-node  system.  The  horizontal-  axis  plots  the  probability  a  given  fault  is  malicious 
for  the  range  from  zero  to  ten  percent.  The  vertical  axis  shows  the  negative  of  the 
common  log  of  the  probability  that  the  faults  present  exceed  what  the  algorithm  is 
rated  to  handle  (so  2.0  equates  to  a  probability  of  one  percent).  It  might  seem  that  for 
most  systems,  probability  malicious  is  small.  This  plot  shows  that  the  system  designer 
cannot  casually  assume  that  probability  malicious  is  negligible.  Of  course,  reliability 
decreases  as  probability  malicious  increases,  even  though  the  plot  shows  the  reliability 
of  the  [Dolev,  et  al]  algorithm  increasing.  We  have  kept  the  expectation  of  [the  number 
of  failures  plus  twice  the  number  of  malicious  failures]  constant  so  that  the  plot  would 
be  better  constrained  on  the  vertical  axis.  So  the  slope  of  the  plot  for  the  [Cristian,  et 
al]  algorithm  is  even  steeper. 

[Dolev,  et  al]  suggested  a  way  of  moderately  reducing  the  number  of  messages 
sent  when  few  faults  are  expected.  We  have  modified  this  algorithm  to  allow  for  a 
variable  reduction  in  message  complexity  and  to  take  advantage  of  the  extra  messages 
sent  by  using  them  to  tolerate  benign  failures.  As  a  result,  we  can  tolerate  a  mixture  of 
benign  and  malicious  failures,  thereby  improving  reliability.  The  reliability  achieved, 
however,  is  slightly  sensitive  to  the  accuracy  of  the  designer’s  estimate  of  probability 
malicious. 

To  overcome  this  sensitivity,  we  have  developed  another  algorithm  (called  mixed- 
sum  algorithm)  that  achieves  still  greater  reliability  and  does  not  depend  on  probability 
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Figure  8:  Reason  for  Considering  Dual  Failure  Modes 
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Figure  9:  Potential  Reliability  Improvement 


malicious.  This  algorithm  is  an  adaptation  of  the  algorithm  given  by  L.  Lamport,  et 
al,  “The  Byzantine  generals  problem,”  ACM  Trans.  Prog.  Lang.  &  Sya.,  vol.  4,  no.  3. 
Our  algorithm  achieves  the  provably  maximal  reliability  under  the  dual  failure  mode 
model.  Figure  9  plots  the  reliability  achieved  by  (1)  the  better  of  the  [Cristian,  et  al] 
or  (Dolev,  et  al]  algorithms  against  (2)  our  mixed-sum  algorithm.  Attention  should  be 
drawn  to  the  significant  improvement  in  the  critical  region  (0  to  10  percent  probability 
of  malicious  failure). 

The  [Lamport,  et  al]  algorithm  has  a  very  large  message  complexity.  While 
our  mixed-sum  algorithm  shares  this  drawback,  we  are  continuing  this  research  to  (1) 
develop  a  more  efficient  mixed-sum  algorithm,  (2)  analyze  other  failure  modes  suggested 
by  our  taxonomy,  and  (3)  consider  bus  networks  (instead  of  point-to-point). 

4.2  Methodologies  for  Designing  Defect-Tolerant,  Reliable, 
Testable  VLSI  Systems  and  Evaluating  their  Costs 

This  research  aims  at  developing  methodologies  for  designing  VLSI  systems  which  are: 
defect-tolerant,  reliable,  testable.  Also  it  evaluates  the  penalty  that  has  to  be  paid 
in  terms  of  area,  performance,  yield,  etc.,  to  achieve  such  properties.  Random  Access 
Memories  (RAMs)  are  to  be  investigated  initially,  because  these  devices  are  the  highest 
density  VLSI  devices  to  be  fabricated  commercially.  Though  they  are  very  difficult  to 
test,  their  regularity  permits  innovation  and  experimentation  and  the  results  obtained 
can  be  extended  to  other,  less  regular  systems.  -  Finally,  because  of  the  low  cost  and 
wide  usage  of  RAMs,  any  architectural  improvements  that  result  in  increased  yield  or 
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enhanced  performance  is  likely  to  have  considerable  practical  significance. 

4.3  Results 

4.3.1  TRAM  Architecture 

A  new  architecture  for  Multi-Megabit  RAMs,  the  Tree  Random  Access  Memory  (TRAM) 
architecture,  has  been  developed.  Applying  the  principle  of  divide  and  conquer,  the 
RAM  is  partitioned  into  modules,  each  appearing  as  the  leaf  node  of  a  binary  inter¬ 
connect  network.  This  network  carries  the  address/data/control  bus  which  permits 
the  nodes  to  communicate  with  the  outside  world  and  with  certain  test  logic  embed¬ 
ded  within  the  chip.  Such  an  architecture  is  shown  to  be  fault-tolerant,  improving 
both  yield  and  reliability.  Also,  it  is  easily  partitionable,  improving  the  probability  of 
generating  partially  good  products.  Parallelism  in  testing,  and  partial  self  test  results 
in  a  dramatic  savings  of  testing  time.  Finally,  unlike  other  fault  tolerant /testability 
schemes,  this  approach  promises  improved  performance  in  terms  of  lower  access  times, 
as  well  as  reduction  in  the  time  required  to  refresh  the  device.  These  benefits  are 
obtained  at  only  a  small  increase  in  chip  area.  These  results  are  obtained  by  detailed 
VLSI  area/ performance  models  that  take  into  account  implementation  and  technology 
dependencies. 


4.3.2  On  chip  error  control  coding  for  yield  and  reliability  enhancements 
in  dynamic  RAMs 

Reduction  in  the  DRAM  cell  size  has  increased  its  susceptibility  to  alpha  particle  ra¬ 
diation.  On  chip  error  detection  and  correction  can  provide  operational  fault  tolerance 
against  these  soft  errors.  This  research  proposes  and  analyzes  two  new  coding  tech¬ 
niques  for  on  chip  ECC:  the  product  code  with  full  code  word  correction  on  each  access 
and  the  odd-weight-column  codes.  Our  proposed  design  differs  from  earlier  designs  in 
its  implementation,  with  the  potential  for  better  performance  as  well  as  better  reliabil¬ 
ity  through  smaller  error  latency.  The  area/ performance  costs  of  implementing  these 
codes  are  analyzed  for  three  RAM  sizes  -  1M,  4M  and  16M  -  and  for  varying  numbers 
of  information  bits  -  from  64  bits  to  2K  bits.  The  analysis  shows  that  the  area  cost  of 
implementing  these  codes  is  low  (<  10%)  for  large  RAMs.  For  each  of  the  RAM  sizes,  it 
also  predicts  the  optimal  number  of  information  bits  for  both  codes  that  will  minimize 
area  and  performance  cost.  Overall,  it  is  seen  that  the  odd-weight-column  codes  have 
a  lower  area  and  performance  cost.  The  analytical  model  used  is  quite  general  and  can 
be  used  to  analyze  the  cost/ performance  of  other  codes,  as  well  as  other  fault/defect 
tolerant  techniques. 

4.4  Work  in  Progress 

1.  For  the  TRAM  architecture,  techniques  to  simplify  restructuring  the  architecture- 
both  during  fabrication  and  in  real  time-Are  being  explored.  These  would  not 
only  simplify  the  generation  of  partially  good  products,  thus  increasing  the  effec- 


tive  yield,  but  also  permit  graceful  degradation  in  the  event  of  real  time  failures. 
Detailed  yield  and  reliability  modelling  is  in  progress. 

2.  A  major  implementation  of  the  TRAM  architecture  is  in  progress.  A  256K  RAM, 
organized  as  16  nodes  of  16K  bits  each,  is  being  designed  in  1.25 nm  CMOS 
technology.  This  will  be  fabricated  by  MOSIS  and  tested. 

3.  Yield  and  reliability  analysis  is  in  progress  for  DRAMs  using  on  chip  error  control 
coding.  An  interesting  possibility  that  is  being  explored  is  the  use  of  the  hardware 
that  is  already  present  for  generating  the  checkbits/syndrome,  for  aiding  off  line 
testing. 

4.  The  TRAM  architecture  is  being  extended  for  Wafer  Scale  Memory  Systems.  Be¬ 
cause  of  its  hierarchical  redundancy,  easy  restructurability  and  the  H-tree  bus 
structure  that  equalizes  the  access  time  to  all  nodes,  this  architecture  is  particu¬ 
larly  suited  for  Wafer  Scale  implementation.  However,  the  degree  of  redundancy, 
and  the  area,  yield,  performance,  testability  tradeoffs  are  very  different  for  WSI 
and  these  are  being  explored. 

5.  The  concepts  developed  for  RAMs  are  being  extended  to  other  parallel  computing 


structures. 
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1.  Copy  of  letter  from  Control  Data  to  RADC 

2.  Copies  of  selected  publications: 

•  D.K.  Pradhan,  “Fault-Tolerant  Multiprocessor  Link  and  Bus  Network  Architec¬ 
tures”,  IEEE  Transactions  on  Computers,  January  1985. 

•  F.J.  Meyer  and  D.K.  Pradhan,  “Dynamic  Testing  Strategy  for  Distributed  Sys¬ 
tems”,  Proc.  FTCS-15,  June  1985 

•  I.  Koren  and  D.K.  Pradhan,  “Yield  and  Performance  Enhancement  through  Re¬ 
dundancy  in  VLSI  and  WSI  Multiprocessor  Systems”,  IEEE  Proceedings,  Vol. 


I 

(§§)CONTRPL  DATA 

V  8800  Queen  Avenue  South 

Qtf  Mailing  Address/Box  1305 

Minneapolis.  Minnesota  55440-1305 

R 


S 

K 

$■ 

I 

! 

I 


January  7,  1987 


Prof.  Dhiraj  K.  Pradhan 
Department  of  Electrical 
and  Computer  Engineering 
University  of  Massachusetts 
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Dear  Prof.  Pradhan: 

Attached  is  a  copy  of  the  letter  I  wrote  to  Mr.  Kaminski  at  RAD C. 
Mr.  Kaminski  is  the  person  you  need  to  contact  for  a  copy  of  the 
Tightly  Coupled  Network  for  VHS1C  Architectures  Final  Report. 


Sincerely, 


Control  Data  Corporation 


/sh 

Attachment 


V 


£ 


.V 


i 


1 


R 

I 

1 


8 
.  ■ 


£ 


s  • 


I 

,  ■* 
.v 

I 


(gg)CONTRPL  DATA 

8800  Queen  Avenue  South 
Mailing  Address/Box  1305 
Minneapolis,  Minnesota  55440-1305 


December  16,  1986 


RADC/COTC 

Griffiss  Air  Force  Base 
Rome,  New  York  13441 

ATTN:  Robert  Kaminski 


Dear  Mr.  Kaminski: 

As  an  author  of  the  Tightly  Coupled  Network  for  VHSIC  Architectures  Final 
Report  that  was  recently  submitted  by  Control  Data,  1  request  that 
Professor  D.  K.  Pradhan  be  provided  a  copy  of  the  report.  The  Professor 
has  intense  interest  in  fault-tolerant  network  architectures,  and  some  of 
the  results  of  his  research  has  been  applied  in  the  TCN  design  concept 
presented  in  that  report. 

He  is  engaged  in  DoD  sponsored  research  in  this  area  under 
Grant  AFS0R  84-0052.  His  mailing  address  is: 

Prof.  Dhiraj  K.  Pradhan 
Department  of  Electrical 
and  Computer  Engineering 
University  of  Massachusetts 
Amherst,  MA  01003 


Sincerely, 


James  W.  Chapman 
Control  Data  Corporation 
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Fault-Tolerant  Multiprocessor  Link  and  Bus  Network 

Architectures 

DH1RAJ  K.  PRADHAN.  shnior  MtMBhR .  tin 


Abstract  —  This  paper  presents  a  general  class  of  regular  net¬ 
works  which  provide  optimal  (near-optimal)  fault  tolerance. 

The  proposed  networks  compare  favorably  to  other  regular 
networks  such  as  leaf-ringed  binary  trees  and  cube  networks.  In 
particular,  the  networks  proposed  possess  certain  advantages  in 
that  the  number  of  connections  per  node  is  neither  an  arbitrarily 
fixed  number  (as  in  leaf-ringed  trees)  nor  does  it  grow  arbitrarily 
large  with  the  size  of  the  network  (as  in  cube  networks).  This  point 
has  significant  relevance  to  fault  tolerance  in  that  the  degree  of 
fault  tolerance  provided  by  the  network  can  be  varied  according 
to  the  design  specification.  Also,  the  networks  admit  simple  self¬ 
routing  of  messages  and  that  routing  is  adaptable  to  faults. 

Index  Terms  —  Algorithmic  routing,  circuit  switching,  con¬ 
nectivity,  diameter  of  graphs,  fault-tolerant  communication  net¬ 
work,  multiple  bus  network,  multiprocessor  networks,  packet 
switching,  regular  graphs,  regular  networks,  shared-bus  fault 
tolerance,  shuffle-exchange  graph. 

I.  Introduction 

RECENT  developments  in  technology  have  made  it  pos¬ 
sible  to  interconnect  a  large  number  of  computing  ele¬ 
ments  in  order  to  form  an  integrated  system.  Various  network 
architectures  have  been  proposed  that  are  suitable  for  both 
multiprocessors  and  VLSI  svstems  1 1 1-|8|.  1 1 1 1-|  14 ] . 

1 19|-[27J. 

The  likelihood  increases  of  one  or  more  elements  failing 
with  the  increasing  number  of  elements  in  the  system.  Con¬ 
sequently,  a  key  consideration  in  the  design  of  such  systems 
is  their  overall  reliability  and  fault  tolerance.  The  fault  toler¬ 
ance  of  a  system  can  be  defined  in  various  ways.  One  mea¬ 
sure  that  possesses  relevance  to  a  system  which  consists  of  a 
large  number  of  homogeneous  elements  is  the  maximum 
number  of  elements  which  can  become  faulty  w  ithout  discon¬ 
necting  the  system.  That  is.  the  assumption  is  made  that  the 
system  can  perform  in  a  degraded  mode  with  loss  of  one  or 
more  components,  as  long  as  the  system  is  fully  connected. 
Also,  more  importantly,  each  element  can  still  be  capable  of 
communicating  with  all  of  the  other  elements  in  the  system 
with  ease,  in  spite  of  the  faults.  In  the  context  of  commu¬ 
nication  delays,  the  performance  degradation  that  is  due  to 
faults  may  be  measured  in  terms  of  the  increase  in  path 
lengths,  and  the  associated  increase  in  the  routing  overhead. 
So.  it  is  not  only  important  that  the  system  remain  fully 
connected,  but  also  that  the  nodes  be  able  to  communicate 
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with  each  other  fairly  easily  —  preferably  with  only  minor 
modification  to  the  original  routing  procedure.  It  is  precisely 
in  this  framework  that  a  class  of  new  fault-tolerant  architec¬ 
tures  has  been  developed  here. 

The  proposed  class  of  networks  is  regular  in  that  all  of  the 
system  nodes  (elements)  possess  the  same  number  of  con¬ 
nections  per  node.  The  proposed  networks  favorably  com¬ 
pare  to  other  regular  networks  such  as  the  binary  cube 
1 1 1 . 1 6 ) .  generalized  hypercube  networks  (13|,  cube- 
connected  cycles  |2|.  leaf-ringed  binary  tree  networks  |3|. 
and  De  Bruijn  graph  networks  |6|.  1 10).  1 14]  as  seen  from 
Table  I. 

In  general,  the  proposed  networks  possess  the  following 
attractive  features. 

1)  Compared  to  other  networks,  the  proposed  networks 
possess  certain  advantages  in  terms  of  their  number  of  con¬ 
nections  per  node.  Specifically,  networks  such  as  the  binary 
cube  require  that  the  number  of  connections  per  node  in¬ 
creases  with  the  number  of  nodes  (whereas  cube-connected 
cycles  and  binary  tree  networks  use  nodes  that  only  have 
three  connections  per  node).  On  the  other  hand,  the  proposed 
network  of  any  arbitrarily  large  size  can  be  built  using  nodes 
w  ith  any  specified  number  of  connections  per  node.  For  ex¬ 
ample.  given  nodes  with  5  connections  per  node,  one  can 
build  a  network  of  256  nodes  (r  =  4,m  =  4).  or  1024  nodes 
(r  =  4,m  =  5).  or  in  general,  any  arbitrarily  large  4"  node 
network. 

2)  The  internode  distances  are  small.  The  maximum  inter¬ 
node  distances  are  proportional  to  only  the  logarithm  of  the 
number  of  nodes,  inversely  ,  to  the  logarithm  of  the  number  of 
connections  per  node. 

3)  More  importantly  ,  the  netw  orks  are  capable  of  maximal 
or  near-maximal  fault  tolerance. 

4)  Degradation  that  is  due  to  an  increase  in  the  routing 
distances  and  communication  overhead  resulting  from  faults 
can  be  lairly  low  .  A  detour  technique  is  presented  that  allows 
the  network  to  degrade  proportionately  to  the  number  of 
faults 

5)  Also  of  interest  is  the  fact  that  the  network  admits 
sell-routing  ot  messages,  both  when  the  network  is  fault  free 
as  well  as  when  it  is  faulty  Self-routing  refers  to  that  ability 
to  route  messages  from  node  to  node  by  using  information 
such  as  destination  address  tag  bits  contained  w  ithin  the  mes¬ 
sage  and  where  intermediate  nodes  perform  no  additional 
computation  tor  routing  T  his  is  possible  if  the  routing  path 
can  be  determined  algorithmically,  without  using  routing 
tables  and  directories 

jsoi  uo  <  i'jxs  ii  1 1 
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The  next  section  presents  the  proposed  network.  Follow  ing 
this.  Section  111  and  Section  IV  develop  the  fault  tolerance  of 
the  networks. 

II.  Proposed  Network 


First,  given  a  graph  structure  one  can  formulate  a  link 
network  and  a  bus  network  based  on  the  graph  structure  as 
described  below.  Let  G  =  (V,E)  be  an  undirected  graph 
where  V’  is  a  set  of  n  vertices  represented  as  0  through  (n  -  1 ). 
The  set  E  represents  the  edges  denoted  as  ( i.j)  where  i  and  j 
are  two  neighboring  nodes  connected  by  (i.j )  in  G.  Let  LA  and 
BA  be  defined  as  two  mappings  of  G  into  a  link  network  and 
a  bus  network,  as  described  below  . 

Let  LA(G)  =  <PE.  C>  where  PE  is  a  set  of  processing  ele¬ 
ments.  represented  as  PE(0).  PE(  1 ).  ■  •  • .  PE(/i  -  1 ),  which 
corresponds  to  the  set  of  vertices  in  G  Let  C  be  the  set  of 
bidirectional  communication  links.  There  is  a  commu¬ 
nication  link  Cli.j )  in  ('  which  connects  PEG)  with  PE< 7)  iff 
(i.j)  f  E. 

Let  BA(G )  =  <fi.PE)  where  B  represents  a  set  of  buses 
defined  as  BL'S(O).  BL'Sl  Bl'Sl/i  -  1 ).  correspond¬ 

ing  to  n  vertices.  The  set  PE  represents  the  set  of  processing 
elements  defined  as  PEl/'.y)  f  PE  iff  (i.  j )  f  E.  The  process¬ 
ing  element  PEl/.j)  is  connected  to  buses  BUSH)  and  BL'Sl./) 
us  shown  in  Fig  I.  Thus,  the  link  architecture  is  obtained 
by  using  the  interpretation  that  vertices  denote  processing 
elements  and  edges  denote  communication  links.  On  the 
other  hand,  the  bus  architecture  is  obtained  by  using  the 
interpretation  that  buses  are  shown  as  vertices  and  computing 
elements  as  edges  in  the  graph.  Thus,  the  number  of  process¬ 
ing  elements  in  BA(G)  is  equal  to  the  number  of  edges  in  G 
Each  processing  element  is  connected  to  two  buses  and  each 
bus  is  connected  to  a  subset  of  the  processing  elements.  (  The 
number  of  processing  elements  connected  to  Bl'St/t  is  equal 
to  the  degree  of  node  1  in  G  )  This  differs  from  the  con¬ 
ventional  multiple-bus  design  where  all  processing  elements 
are  connected  to  all  buses.  Since  each  bus  is  connected  to  a 
subset  of  processing  elements,  an  inter-PE  transfer  may  re¬ 
quire  several  interbus  transfers 

However,  if  bus  load  is  equated  to  the  number  ol  con¬ 
nections  per  bus.  then  the  WAiGi  network  has  a  much  smaller 
bus  load  when  compared  to  an  equivalent  design  which  uses 
conventional  shared  buses  I  heretore.  a  WA1G1  l\pe  bus  net 


baCC) 

Fig.  1  Link  and  bus  architecture. 

work  may  have  certain  advantages  over  the  conventional 
shared  multiple-bus  design  when  there  arc  large  numbers  of 
processing  elements  to  be  connected.  Also,  it  may  be  noted 
that  one  of  the  advantages  of  a  BAiG)  network  over  MiG) 
network  is  that  it  can  be  easily  extended  by  adding  more  PE's 
to  buses  as  required  and  there  are  well  developed  bus  proto¬ 
cols  available. 

This  alternate  multiprocessor  multibus  architecture  can  be 
quite  attractive  where 

1)  a  processor  may  not  have  the  hardware  capabilities 
to  allow  its  attachment  to  more  than  a  certain  number  of 
buses,  and 

2)  for  reasons  of  reliability,  the  buses  may  be  in  physically 
different  locations:  hence,  a  processor  may  not  be  located  next 
to  every  bus. 

Various  relationships  between  G  and  the  corresponding 
LA(G)  and  BAiG)  are  described  in  Table  II  In  describing  the 
FG  networks  below  first  the  underlying  graph  structure  EG  is 
defined. 

FG  Network  Design:  The  number  of  nodes  in  the  graphs 
defined  below  is  assumed  to  be  equal  to  rm.  As  seen  later,  the 
chosen  values  of  r  and  m  will  determine  the  number  of  con¬ 
nections  per  PE  lor  BI  S),  the  routing  distance  between  PE's, 
and  the  degree  of  fault  tolerance. 

Here,  the  nodes  are  assumed  to  be  numbered  0  through 
in  -  li  Each  node  /  has  an  m-tuple  representation  in  radix-r; 
this  will  be  denoted  as  1/.,  ■  .i.i.,). 
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TABLH  II 

C*i  r r  ms  IXh  in  \i  i  v  i  s  bitnm  kn  BArti f  Ni  r\U)RKs  and  LAiG)  Networks 


LMC  i  link  network 

Equivalence  in 

BMC  i  bus  network 

Numher  of  processors 

Number  of  buses 

Number  of  links 

Number  of  processors 

Numher  ot  connections- prinressors 

Number  of  connections  bus 

Interprocessor  transfer 

Interbus  transfer 

Number  of  processor- processor 

Number  of  bus- bus 

transfers  in  a  message  path 

transfers  in  a  message  path 

Processor  fault 

Bus  fault 

Link  fault 

Processor  fault 

exchange  mappings,  respectively  ( 8 1 .  [9|.  1 18|). 

Thus.  I  =  /i(  /)  implies  /  =  /*(/>.  i  and  j  differ  only  in  the 
last  digit. 

These  graphs  are  constructed  using  a  two-step  approach. 
First,  a  skeletal  graph  SG  is  constructed,  which  is  then  aug¬ 
mented  to  obtain  the  fault-tolerant  graph  FG. 

The  skeletal  graph  denoted  as  5 G(r.  m )  is  obtained  by  con¬ 
necting  every  pair  of  nodes  /  and  j  that  satisfy  the  relationship 
i  =  g(  j)  or  /'  =  hi  j). 

In  the  following,  let  A  =  ( r”  -  1 )/( r  -  1).  Thus,  k  = 
( I .  I .  •  •  ■  I )  in  radix-r. 

FGlr.  mi  Design,  m  ~  2: 

r  =  even:  Construct  an  SG ( r.  2)  graph  for  the  specific 
r:  then  augment  the  graph  by  adding  r/2  links,  defined  as 
(0.  A  ).( 2A.  3 A  (tr  -  2 )A.  (/•  —  I  )A  ). 

Fig.  2  illustrates  an  FG(4.  2)  network. 

r  =  odd:  Construct  an  SGir.2)  graph  for  the 
specific  r ;  then  augment  it  by  adding  an  extra  node  n  which 
is  then  connected  to  nodes  0.  A.2A.  ••  •.(/•  -  I  lA  bv  adding 
r  additional  links:  t/t.  0),  in.  A  ).••  •.(//.  tr  -  I  )A  ).' 

(The  resulting  graph  when  r  -■  odd  has  in  -  li  nodes. 
This  extra  node  can  be  used  as  a  spare  and  as  shown  later,  is 
useful  for  routing  when  faults  occur  in  the  system. I 

FGlr.  mi  Design,  m  •  2: 

( ir"'  -  li.tr;  —  1 1  for  m  even. 

Let  q  =  \ 

I  ir"  -  1  I  lr  -  I  l  lor  m  -  odd  . 


Thus,  in  radix  r, 

(0.  1 . 0.  1 .  ■  ■  ■  .  0.  1 )  lor  m  =  even 

(/  = 

1  I  .  0.  I  (I.  •  ■  ■  .  I  .  (I.  I  l  lor  m  odd 

r  =  2:  Construct  an  SGiZ.mi  graph  for  the  given  m. 
then  add  links:  (0,  A  i.  ( A .  A  <y t .  and  i </.  Ui  For  odd  m  add  an 


It  ma\  he  noted  that  there  cannot  e\r*»f  am  decree  •  regular  graph  ol  e\.ull\ 
r  nodes  since  r  <»dd  Therefore,  one  rimo  ad.!  .n  additional  node 


additional  link  (A  -  q.q). 

FG(2.4)  graph  is  illustrated  in  Fig.  3. 
r  g  4.  Construct  an  SG  ir.  in)  graph  for  the  given  r.  m . 
Add  r  links,  as  defined  below  :  (0.A).  ( A .  2A 
(ir  -  2)A.  (r  -  l)A),((r  -  llA.0).  Ifm  =  even,  add  addi¬ 
tional  (/••  -  r)/ 2  links  defined  by  the  following  expression 
for  all  a.h.  a  b  and  0  a.  b  v_  ir  -  I ): 

itirq  -  bcf.ir  -  I  -  alrq  +  ir  -  I  -  h)q 1. 

Fig.  4  illustrates  FGi 3.3)  graph. 

The  following  basic  properties  of  FGir.  m)  networks  can 
be  easily  proven. 

Theorem  I:  FGir.m)  is  a  regular  network  of  degree  r  it 
m  =2. 

Theorem  2:  FGlr.  ml  is  a  regular  network  of  degree 
(/'  -  h  if  r  2  3  and  m  .:  3. 

Theorem  .<  FGil.ml.m  '  3.  mi  =  even  are  regular  net¬ 
works  of  degree  3.  For  m  ~  odd.  all  nodes  are  of  degree  3 
except  nodes  q  and  (A  -  q).  which  have  degree  5. 

Thus,  the  I.AiFG  I  and  HA  (FG  )  networks  derived  Irom  FG 
will  have  the  following  characteristics  F.ach  PF.  in  l.Atl  G  i 
will  have  the  same  number  of  links  connected  to  it  eithei 
r  or  ( r  ■  h  Analogously,  each  bus  in  HAiFG  I  will  be  con¬ 
nected  to  the  same  number  o!  PF  s  either  r  or  i r  •  li 
I  hus.  Irom  the  point  ol  view  ol  I  ()  ports  and  inter- 
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f-ii!  5  Routing  on  bus  architecture 


connections,  these  networks  can  he  considered  regular  The 
similarities  and  differences  are  noted  that  exist  between  the 
link  network  and  the  bus  network,  are  described  in  Table  ii 

The  following  shows  that  a  path  always  exists  of  length, 
at  most  (2m  -  I ).  given  any  pair  of  nodes  in  FGtr.mt 
networks. 

Tirst  it  may  be  noted  that  in  the  link  architecture  I.A(FG  t. 
transmitting  data  from  one  PH  to  another  PH.  may  require  one 
or  more  hops  through  other  PH's  m  the  path  Similarly,  in  the 
bus  architectures  BA(FG).  direct  translcr  is  possible  only 
when  both  ot  the  processing  elements  are  connected  to  the 
same  bus.  i.e  ,  PH.l  <.  \ )  can  transfer  to  PH.irr.  m  if  r  «  or 
i'  or  if  v  =  i<  or  o  In  other  cases,  a  transfer  would  require  one 
or  more  interbus  transfer  through  the  connecting  PH's  How¬ 
ever.  the  dilterence  between  the  bus  architecture  and  the  link 
architecture,  insofar  as  formulating  a  routing  path  between  a 
source  and  a  destination  PH  is  concerned,  is  that  in  the  bus 
architecture  various  choices  exist  depending  on  the  various 
combinations  of  the  source  and  destination  buses  H'or  ex¬ 
ample.  given  PKtt.  v)  as  the  source  and  PHlw.in  as  the  des¬ 
tination.  PHf.tr.  >i  can  initiate  the  transmission  on  bus  r  or 
v.  and  PH(//.  v)  can  receive  it  from  bus  u  or  v  as  illustrated 
in  Fig  5 

However,  topologically,  there  are  certain  equivalences  be¬ 
tween  paths  in  l.A(Ki  I  and  BAiFG  ).  Given  a  PH-to-PH  path 
in  the  link  architecture,  there  is  an  equivalent  HI  S-to  BI  S 


path  in  the  bus  architecture  II  an  inter  PH  transter  constitutes 
a  single  hop.  both  ot  these  paths  will  have  the  same  number 
of  hops  On  the  other  hand,  given  a  PH.  to-PH  path  in 
BA(FG\.  there  is  an  equivalent  link  to  the  link  path  in 
l.AiHi  i  Below,  the  routing  in  these  networks  o  described 
in  the  context  ot  the  link  network  l.Atf-Gt  Ihe  formula¬ 
tion  below  can  be  adapted  tor  the  bus  architecture  BAtHi  i 
as  well 

Consider  the  following  paih  Irom  the  source  PH. i  s  >  to  the 
destination  PH. ( </ 1  l  et  v  n.  \  .  v  i  and 

ii  til...  .</.</. i  in  radix -r 

S  I  .  .  \  .  \  1 

l  s  „  .  .  s  .  \  .  s ...  I 

I  V...  .  .  \  .  V  .  it..  I 

Iv.  .  v  .,/  V.  I 

IV  .</..  .,/ . 1 1  I 

hL  .  .  il  .  v  i 

,/  !</...  .  .</.</  I 

The  above  path  w  ill  be  herew  ith  denoted  as  pi t  \  d<  I  ho 
is  used  below  to  formulate  a  simple  message  routing  prose 
dure  that  mutes  the  message  from  mule  to  node  using  onlv  ihc 
destination  address  information 

A  message  routing  algorithm  suitable  lot  /  ft  /  (< 1  >  m  <  >  o 
described  here  It  is  assumed  that  each  message  sarries  m  tag 
bits  These  m  bits  denoted  as  /  are  imtiali/ed  at  the  soutsc 
PH.  equal  to  «/  the  destination  address  as  shown  m  1  ig  hi  a 
and  ibi  The  destination  addre"  ■/  is  also  s  a r r t e v I  hv  the 
message  separately 

Although  the  routing  algorithms  given  below  ate  tot 
I.A(Ki)  networks,  thev  van  he  toed  tor  the  hu'  network 
BAiFG  i  as  well  with  some  moditieations  1  ,k  h  PH i »  '  m 
I.A( Ki  l  corresponds  to  a  node  i  in  Hi  and  vise  versa  Also 
every  path  from  PH. i  1 1  to  PHi  v  i  m  /  fi  h  G  i  has  a  v  orre'pon.l 
ing  path  Irom  node  >  to  v  m  Hi  and  vkc  versa  Iheretore 
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.ill  the  path'  and  rout  mi;  us  turns  t*»r  l  At  b  (»  i  will  be  described 
hs  using  the  graph  ht,  Hie  remainder  ol  the  section  will  use 
node  i  to  denote  Phi  t 1 * * * S 

XV  hen  .1  message  arrises  at  node  v .  the  following  steps  are 
evesuted  to  determine  the  next  node  in  the  path  Here.  < 
denotes  the  least  significant  hit  of  the  bmurs  number  l 
x/(  r  hi  i/.  then  the  message  has  reached  the  desti¬ 
nation  and  ts  assepted  Otherwise  the  message  is  tsirw arded 
to  a  neighbor  o|  i  hs  usme  the  lolloviine  steps 

S ; ,  />  2  Compare  >  'the  least  'igmticunt  hit  of  ii  to 
'he  'eadme  h*ii  ■  >!  /  It  thes  are  equal  then  go  to  Step  V 
else  !■  a  a  at  d  the  message  to  the  neighbor  ol  i.  gisen  as 
i  .  >  i  ■  m  hours 

V.  r  '  Shot  the  tag  field  /  left  hs  >nc  hit  as  shown  in 
I  c  |  Non  t  i  o  oi  i  i  n  I  i  then  go  to  Step  2. 
else  I  or*  aid  Uv.  message  lo  the  neighbor  .  of  i.  gisen  as 
i  i  i  i  i  m  h.nats 

I  In  ton, .a  nc  .  sample  illustrates  the  abuse  touting  steps 
'  atthi  ; 

/  1,1"!,"',  V  otisidct  the  lh  node  Hi  1  2  4>  neissorts  l  et 
s  and  /  I!  Ills  following  table  describes  the  path 
md  'In  .  o; ,iu!ing  tag  ’’it'  at  dillerent  nodes 

s  o  ;  ;  i  '  in  1 1 

ii  ii  i  ,  ii  i  |  ,  ii  i  |  |  ,  ii  i  !  i  i  i  i  1  ii  i  i  i  |  i  i  |  |  i  i  |  i  i  |  I  i  |  | 

oil  I  ,  I  III  III  III  |  1 1  ii  i  I  1 1  II  i  1 1  mm  i  |  Ml  M  i 

O  i  I  i  |  I  i  i  ,  i  I  I  i  ,  I  |  I  II  I  |  |  I  II  I  |  I  1!  II  I  I  I  H  H  I 

i  't  it  d'o,,  steps  in. ite  the  message 

;  i'n  pt  s  </  dc'sidvd  it  icr  I  he  routine 
!  i’n  n  .mi  i'Iit.  ..  md  i'i  s  onls  Im  a! 

;  I  'I,  -I,  S..I.V 

III  I  I  -  •  I  (  I  :  ...  V  A  ak  . 

•  '  _  '  'I  Oli  'O  ..III!  ,||S 

'  .s  ■  s  I  ■  '  It,  ■>■  •  net 

•It,  ■  ■  '  s  ■  ►  -  lot  an  . 

■It,  ■'  ,  .  •  a  i  •  tie  de 

■  - . i c .  -  a  ' ' ■  i  ■  i  I'gle  and 

S  \\  a  -  dc-  /  <1 

1 "  "  -  '  '  -i  -  1  i'i '!  "' 

O'  '  '  |  r.  . .  ll  _ 

'  i  1  i  o'  '  tig  Malls 

' '  ■ 1 ' '  i  , '  i  c  '  ’  a  U  C  \ 

I.  .  ■  id  •  . .  ‘I  ,i'.d,  n  ■ 


No,!; 

r  ’ 1 

/  I-  I"  I 


17 


In  considering  the  fault  tolerance  ol  LAit'G  )  and  BAiFG  ). 
the  following  mas  he  noted  regarding  the  effects  of  various 
faults  from  the  point  of  stew  of  communication  and  routing, 
as  described  tn  Table  II  The  effect  of  a  faults  PE  tn  the  link 
architecture  is  equivalent  to  the  effect  of  a  faulty  BUS  in  the 
corresponding  bus  architecture  Similarly,  a  faults  PE  in  the 
bus  architecture  has  an  equivalent  effect  on  the  routing,  as  a 
faults  link  in  the  link  architecture  The  following  discusses 
tauit  tolerance  in  the  context  of  link  architecture  and  this  can 
lairls  easily  be  extended  to  bus  architecture 

Primarily,  node  failures  are  considered  here  since  the  ef¬ 
fect  of  a  link  failure  can  be  no  worse  than  a  node  failure.  (The 
paths  affected  hs  a  link  failure  are  a  subset  of  the  paths 
atteeted  by  the  tailure  of  one  of  the  nodes  connected  by  the 
link  Therefore,  the  routing  and  detour  techniques  can  be 
adapted  to  link  latlures  easily  ) 

The  fault  tolerance  of  a  regular  network  of  degree  b  can 
he.  at  most,  equal  to  th  -  I )  This  is  because  any  node  can 
always  be  disconnected  from  the  network  by  removing  the  b 
nodes  1 1  ink's  *  that  are  connected  to  it.  Accordingly,  sse  will 
call  a  regular  network  an  optimal  fault-tolerant  network  if  its 
tault  tolerance  is  equal  to  t b  -  ll.  the  maximum  possible. 

Later.  Khl.nn  networks  are  shown  to  be  optimally  fault 
tolerant  < m  -  event 

The  case  of  single  faults  is  considered  separately  first.  An 
algorithm  is  developed  below  that  routes  messages  from  one 
node  to  another,  in  spite  of  any  single  faulty  node  or  link 
(liven  a  binary  number  i.  let  the  weight  of  i.  Iwtlxll. 
represent  the  number  ol  I  s  in  s 

Tcnnmi  I  Any  node  \  tn  the  path  pt  ti.fU  satisfies  ti  e 
relationship  given  below 
for  i  r  i. 

nil  u  UP/I.  ll  I,  -  0 

M  It  l  I  ll/lj!.  it  i  I 

Fro,  t  Consider  ans  two  consecutive  nodes  u  and  i*  in 
the  path  p:  1 1 .  (I  i  |  ct  n  t  it.,  ■  •  •  .  it  .it  I  and  r 
m  .  r  .  i'  i  in  binary .  ansi  let  n  precede  r  in  the  path 

I  he  follow  ing  relationship  between  it  and  r  must  he  satis- 
ticsl  i  is  obtaincsl  Irom  it  bs  an  end-around  shitt  ol  it  it  the 
least  sigmlts.int  bit  « > I  n  is  o  or  bs  complementing  the  leasl- 
signitKuni  bit  nt  a.  it  the  least  significant  hit  ot  it  is  I  I  hits, 
the  number  "I  I  A  m  r  sannot  exseed  the  number  ot  I  A  in  it 
I  hi  the i  more  it  u  I.  then  the  number  ol  I  A  in  r  is  less 
than  tbs'  number  <>t  I  s  m  it 

liens e  the  I  emnia  (,*1  U 

I  ,  mm,,  2  'ns  in  isle  i  in  the  path  />n  / .  ii  I'  sati'ties 
1  tis  lull. us  me 
I  •  a  i  t  ■ 

„  i  ■  ■  a  r  ■  •  it;  I 

i  \ !  ■  i  1  1 1  / 1  /  *  1 1  /  n 


F . •  Ptiml  is  sinnl.it  t.i  I  emma  I  O  1  I) 

I  tis  pi , i.  *!  -  ■'!  the  I •  ■  1 1 ■  •  vs  i r i g-  I  ennuis  ,ue  als,.  sunilat 
!,•••! >ii,  .  \ir,  ii,,.ls  i  in  ttis  path  pt iii.M  also  sati'lies 
i  ■  :  I 

/  ’  ”,  J  \iis  ii.  ide  i  m  tlis  path  pt  tn  I  ;  i  also  satis 

' u'  I  ,  .''im.i  2 

Ii:.  i  ,'"i  'im  .!,s,  i|ms,|  lie  I,  hs  .  ha  sol  on  the  above  ob 
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servations  and  is  useful  for  routing  messages  when  a  node 
becomes  faulty.  This  also  establishes  an  upper  bound  on  the 
maximum  path  length,  in  the  presence  of  a  fault.  The  algo¬ 
rithm  given  is  shown  to  be  easily  implementable. 

Let /  denote  the  faulty  PH,  PE(/). 

Let  s  denote  the  source  PH,  PE(.v). 

Let  d  denote  the  destination  PE.  PEL/). 

Routing  with  a  Faulty  PE:  The  follow  ing  describes  differ¬ 
ent  paths  from  s  to  d  in  FG  which  correspond  to  different 
cases  off. 

a)  vef(/)  >  vert.v)  and  vert/)  >  wt(d). 

S  — »  ■  ■  ptis,  0)  •••■*■»  o  —*  •  •  ‘  ptiO.  d  )■••—*  d  . 

b)  vert/)  <  urt ,v)  and  wtif)  <  vertc/l. 

s  — ►  •  ■  ■  ptis,  n  -  I  )■••—*  (h  -  1 1  — * 

■■■  ptin  —  1 .  d  )■■■—>  d  . 

c)  vert/)  >  vert.v)  and  vert/)  <  vertc/l. 

,v  — *  •  •  ptis.  0)  •  •  •  — *  0  — »  t/i  —  1 )  — * 

■ptin  -  1 .  d  )•••—»  d . 

d)  vert/)  <  vertc/l  and  vert/)  >  vertc/l. 

s  —*■■■  pUs.  it  -  1  )•••—»( n  -  1 )  — * 

0  — *  •  •  -ptiO.d) - *  d. 

e)  vert/)  =  vert.v)  and  vert/)  >  vertc/). 

(  1  path  as  a  ) 

If  ad  = 

1 0  path  as  c/l  . 

f)  vet/)  =  vert.v)  and  vert/)  <  vertc/). 

(  I  path  as  c ) 

If  v„  =  t  , 

1 0  path  as  h  ) . 

g)  vert/)  =  vertc/)  and  vert/)  >  vert.v). 

I  I  path  as  a  ) 

If  c/„  = 

It)  path  as  c). 

hi  wtif)  =  wrtc/l  and  vert/)  <  verts). 

[  1  path  as  c/ ) 

If  c/,  = 

1  0  path  as  h  I  . 

l)  vert  f  I  =  vert  v)  =  vert  il  I 


w  hich  describes  the  follow  ing  path  from  v  to  d : 
s  — »  •  •  ptis.n  -  I  )•-■—►(«  -  I  )—»■••  ptin  -  1 ,  c/  ) 
- ►  c/. 

This  can  be  implemented  as  follows.  The  source  v  may 
initialize  the  tagfield  T  equal  to  t/i  -  I ).  This  will  route  the 
message  to  the  intermediate  destination  node  in  -  I).  The 
node  in  -  I ),  upon  recognizing  that  the  destination  address 
d  is  not  equal  to  (n  -  1),  will  replace  T  by  d  and  then 
forward  the  message  to  the  final  destination  d. 

Next,  we  consider  the  case  of  double  faults  in  FG(2,m) 
networks  m  >  3.  In  Appendix  A,  a  technique  is  exhibited 
that  can  route  a  message  from  a  source  node  s  to  a  destination 
node  d.  despite  two  faulty  nodes.  The  following  theorem  is 
a  direct  consequence  of  this. 

Theorem  5:  In  the  presence  of  any  two  faults,  a  message 
can  be  routed  from  any  node  v  to  any  node  d  in  FGil.m). 
using  at  most  (4m  +  2)  hops. 

Thus,  a  second  fault  may  cause  a  small  increase  in  the  path 
length;  (4m  +  2)  versus  (4m  -  1). 

IV.  Fallt  Tolerance  ok  FGir.m)  Networks 

This  section  considers  FGir.m)  networks  for  all  r.m  2  3. 
Techniques  are  first  formulated  that  construct  detours  around 
the  faulty  nodes.  These  detour  techniques  are  applicable 
when  the  number  of  faults  does  not  exceed  ir  -  1  )/2.  Fol¬ 
lowing  this,  it  is  shown  that  these  networks  in  general  can 
also  tolerate  a  much  larger  number  of  faults  by  show  ing  how 
to  construct  paths  from  s  to  d  w  hen  the  number  of  faults  is 
equal  to  ir  -  I ). 

The  detour  techniques  shown  below  possess  certain  attrac¬ 
tive  features.  These  detours  perform  local  alterations  of  paths 
which  connect  the  two  nodes  that  are  adjacent  to  the  faulty 
node(s).  Hence,  the  faults  can  be  made  transparent  to  the 
global  routing  strategy.  Other  important  aspects  of  these 
detours  include  the  fact  that  they  are  of  constant  length,  in¬ 
dependent  of  m.  thus,  the  size  of  the  network.  Furthermore, 
increases  in  path  length  that  are  due  to  the  use  of  detours 
become  directly  proportional  to  the  number  of  faults;  thus, 
graceful  degradation  is  made  possible 

Definition  Let  F  =  {/'./'.•••./  }  represent  the  set  of  t 
faulty  nodes. 

Definition:  Let  the  m-tuple  i  I,.  .  I ..  ■.  ■././)  repre¬ 

sent  the  faulty  node  f  in  radix-v 

Definition:  Let  <•  be  a  radix-/'  digit  that  does  not  belong  to 
the  set 


v. ,  ~  0.  </,,  =  0  path  as  h) 

a  =  0.  </  -  I  path  as  d ) 

v,,  =  I .  d,  =  0  path  as  r ) 

s.  --  l .  </.  -  1  path  as  (i  1  . 

The  following  Theorem  is  an  immediate  consequence  of 
the  above  routing  steps 

Theorem  4  In  the  presence  <>!  any  single  fault,  a  message 
win  he  routed  from  anv  node  to  any  other  node  using  at  most 


Thus,  e  is  a  digit  that  does  not  appear  either  in  the  least-  or 
in  the  most-signit leant  position  ot  any  of  the  faulty  nodes 
Since  2t  <.  r.  there  alwavs  exists  such  an  e  I  bus.  any  node 
that  has  c  in  the  least  or  in  the  most  significant  position 
cannot  be  faulty.  This  will  be  quite  useful  later 

Here,  in  constructing  these  detours,  it  will  be  assumed  that 
ii.  v.  and  u  are  three  consecutive  nodes  in  the  path  from  '  to 
(/  The  node  r  is  assumed  to  be  faulty  with/v  and  w  being  fault 


i4m  -  h  hops  free.  The  detour  thcrelore  connects  m  \v  ith  vi  without  passing 

The  earlier  described  routine  procedure  is  easily  adaptable  through  any  ol  the  faulty  nodes  in  f  including  r  <  I  he  case 
when  the  network  become-  tauitv  As  an  example,  assume  when  two  or  more  consecutive  nodes  are  faulty  can  be  treated 
node  ll  is  the  tauitv  node  I  hi-  will  correspond  to  case  b),  bv  successive  applications  ol  the  given  techniques  i 
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First,  it  may  be  seen  that  there  are,  altogether,  three  pos¬ 
sible  distinct  relationships  between  u,  v ,  and  w  in  the  type  of 
path  discussed  earlier  in  the  section.  (This  follows  from  the 
observation  that  if  v  is  an  /i-neighborof  u.  then  w  cannot  also 
be  an  /i-neighbor  of  v. )  These  relationships  are  described  in 
the  following: 

Rl :  v  =  h{u)  and  vv  =  g(u) 

R2:  v  =  g(u)  and  vv  =  h(v) 

R3 :  v  =  g(n)  and  vv  =  g(u) . 

The  following  constructs  detours  that  correspond  to  the 
above  three  cases.  The  detailed  construction  is  described  in 
Appendix  B 

Case  Rl :  Here,  the  original  path  contains  the  following 
sequence  of  nodes  represented  in  radix-r  as 

M  =  \U„]. . 

faulty:  v  =  Im„  . .  m,.c).  c  *=  u0 

vv  =  Im„-:.  ■  •  ■  .  M|.r,  u,„-|) . 


The  following  describes  construction  of  an  alternate  path 
from  u  to  vc  that  does  not  pass  through  any  of  the  nodes  in 
fault  set  F  (which  includes  the  faulty  node  v): 


u  =  lu„  ■  - 

•  •  •  ■  .  K|.  Mu) 

(«„  ■  • 

•  .  M,,.t,  M„  ,) 

x  as  determined  per 

Appendix  B 

‘  ‘ 

•  •  ■  ,  M,.  .r,  e) 

(e,un 

. .  u,,.r) 

e  as  defined  earlier 

1  e.u„ 

. ,  M,.  t  ) 

■  ■ 

•  •  • .  M,.  c.  e) 

vv  =  ■  • 

’  ,  M|.C.M„ ,  ,)  . 

The  above  detour  is  of  length  7,  and  hence  will  result  in  a 

net  increase  of  5 

in  the  path  length. 

Case  R2:  u  — * 

v  =  g(u)  — *  1 

>c  =  h(v).  This  corresponds 

to  the  following  path  segment: 

a  = 

<Mm-|. . 

■  .  M|.  Mo) 

faulty:  a  = 

•  Mo,  M,„  ,) 

VV  — 

(n„  . . 

M|,  M„.  <  )  (  Um  ,  . 

The  following  constructs  an 

alternate  path  from  u  to  vv. 

u  =  lum  |,  • 

. .  M|.  M,,t 

. .  M  | .  X ) 

r  determined  as  per 

Appendix  B 

•  '  . « , .  r ,  u„  ,1 

( u*,  ■ 

•  •  •  •  .  m  i .  x .  e ) 

e  defined  earlier 

1 e . : 

. . .  m  , .  r  I 

(e.  u„  : 

.•••.  M,.  M„) 

(  Ur*  ■ 

■  ■  ■  .11 1  .  Mtl.  fl 

W  =  (M„  • 

’  •  '  .  Mi.  Mo.  (  ) 

TABLE  ill 

Fault-Tolerant  Properties  ok  FG(r.m)  Networks 


r 

m 

Faull 

tolerance 

i 

Routing 

Distance 
with  /  faults 

1 

(4m  -  1) 

IV  V 

2 

m  ^  3 

“) 

(r  -  1) 
l  %  (r  -  1  )/2 

(4m  +  2| 

12 

1 2m  -  1  i-  11/) 

r  >  (r  -  ll/2 

(6m  -  3) 

As  in  Case  R 1 ,  the  detour  shown  above  is  of  length  7,  and 
thus  the  path  length  will  increase  by  at  most  5. 

Case  R3:  u  — »  v  =  g(u )  — »  w  =  g(v).  This  corresponds 
to  the  following  segment: 

u  =  . .  u,.uu) 

faulty:  v  =  (u„-:, . . u„.  m,„-,) 

w  =  [um- m„.  m„  . 

One  can  construct  a  detour  from  u  to  tv.  as  illustrated  in  the 
following: 


(Mrtl  -  \  .  ■  ■  ■ 

. .  M,.  Mo) 

(m„-  1,  •  •  ' 

. .  M|..r) 

r  and  y  determined  as 
per  Appendix  B 

(M,„.;.  •  •  ' 

•  •  .  M|..r, m„.  ,) 

(Mm-;.  •  •  ’ 

. .Mi.JT,  y) 

(m,„.  • 

•  ,M|..r,  V.Mm-,) 

(M„.,,  '  '  • 

'  ■  •  ,M|..t.V,fl 

e  defined  earlier 

(e.  um.  o  • 

. ,  M|..v,  v) 

(e.  u„  u  ■ 

. .  M|..v,  e) 

(e.  e.  m,„.  i 

i. . .  m,.jt) 

(e.e.u,„  x 

i, . .Ml  Mo) 

(e.  um  x.  ■ 

. .  Mo.  e) 

(e.  um  x.  ■ 

•  ■  •  ■  .  Mo.  Mm  I ) 

ium 

•  ’  .  Mo.  M,„  |,  f) 

( M.„  •  •  • 

.  Mo.  It.,.  | .  M,„  ;)  . 

The  above  detour  is  of  length  13:  thus,  it  w  ill  result  in  a  net 
increase  of  1 1  in  the  path  length. 

The  above  corresponds  to  the  worst  case  increase  in  the 
path  length  It  may  be  noted  that  when  two  or  more  con¬ 
secutive  nodes  are  faulty,  the  above  techniques  can  be  ap¬ 
plied  iteratively,  to  construct  a  detour  around  the  faulty 
nodes.  As  an  example,  consider  the  following  path  segment: 

u  — *  v  —  h(u)  — »  vc  =  g(u)  —i ►  r  =  g(w) . 

Assume  here  that  both  the  nodes  v  and  w  are  faulty.  Thus, 
one  needs  to  construct  a  detour  from  a  to  r.  This  can  be 
constructed  in  two  steps:  first,  by  using  Case  Rl.  one  can 
construct  a  detour  of  length  6  from  u  to  ( u„  ■  •  •  ,«],<  .<•) 

Next,  by  applying  Case  R2,  one  can  construct  a  detour  of 
length  1 1  from  (um  ••  •  .  it, ,  r.  <>)  to  :  =  lu„  4.  •  c, 

u„  i,u„  :.u„  ,)  Thus,  the  total  detour  length  will  be  17  for 
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both  the  faulty  nodes  together.  The  following  theorem  is  a 
direct  consequence  of  these  above  discussions. 

Theorem  6:  A  message  can  be  routed  from  one  node  to 
another  using  at  most  (2m  -  1  +  11/)  hops  with  /  faults 
where  /<(/•-!  )/2. 

The  following  in  hibits  the  complete  fault  tolerance  of 
FG(r,  m)  networks  for  all  r,  m.  r  s  3  m  >  2.  It  is  show  n  that 
in  spite  of  any  (r  -  1)  faults,  the  network  remains  fully 
connected  and  the  messages  can  still  be  routed  easily  from 
node  to  node. 

Definition:  Let  N  denote  the  set  of  all  n  nodes  in 
FG  ( r.m ) . 

Definition :  Let  Rt  denote  the  set  of  r  nodes,  0,k.2k. 
}k.  ■  •  ■  .  (r  -  1  )£. 

Definition:  Let  N  —  Rk  denote  the  set  of  (n  -  r )  nodes 
consisting  of  all  nodes  other  than  those  appearing  in  the  set 

Rk. 

Fnult  Tolerance  of  FG(r,  ml,  m  =  2.  Networks:  Since 
m  =2,  any  node  i  in  the  network  can  be  represented  in 
radix-/-  as  (/,.  Now,  consider  the  following  r  paths  from 
s  to  the  nodes  in  Rk. 

v  to  0.  ,v  =  (.?,.*„)  -<•  (.9, . 0)  —  (0.5,)  —  (0,0)  =  0 

5  to  k:  s  =  ( .v, ,  ,vtl)  — *  ( .v i .  I )  — *  ( 1 . 5|)  — »  (1,1)  =  k 

s  to  2k:  s  =  ( 5 , . .A',, )  — »  (.v,. 2)  — >  (2.5,)— »  (2,2)  =  2k 

.v  to  (r  -  1  )k:  s  =  (,V|,5„)— *  ( s,.r  -  1) 

—*  (r  -  1 .  .v,)  — >  (r  -  1 .  r  -  I ) 

=  (r  -  \)k. 

These  above  paths  can  be  seen  to  be  disjoint  from  the 
following  observation. 

Assume  that  there  exists  a  node  in  the  path  from  v  to  ik  that 
also  belongs  to  the  path  from  s  to  jk  w  here  i  *=  j.  This  would 
imply  either  that  (.v,, /)  =  ( j.x,).  or  that  < .v, , /)  =  ((.  a,).  So. 
one  has  ,V|  =  i  =  j.  a  contradiction. 

It  may  therefore  be  deduced  that  there  always  exists  a  node 
xk  for  which  the  path  s  to  xk  is  fault  free,  in  spite  of  any 
(r  -  I )  faults. 

Similarly,  it  can  also  be  asserted  that  there  always  exists  a 
node  xk  for  which  the  path  xk  to  d  is  fault  free,  in  spite  of  any 
(r  -  I )  faults.  So.  in  order  to  establish  that  the  fault  toler¬ 
ance  of  the  network  is  (r  -  I ),  it  w  ill  he  sufficient  to  show 
that  there  always  exist  r  disjoint  paths  between  any  xk  and  xk. 
The  following  shows  techniques  to  construct  such  /  paths 
from  xk  to  xk. 

(The  paths  are  shown  here  using  rudix-r  representation 
Thus,  xk  =  ( .v ,  v (  and  xk  —  (  i.v)  in  radix-/.) 

First,  consider  the  following  set  of  (/•  -  1 1  paths  denoted 
as  path  I  -path  ir  -  1 ). 

path- 1 :  f  t  .  v)  — »  I  v.  vl  — *  (  v.  1 1  — *  l  v.  i  I. 

path-2-path  tr  -  1 1:  Let  it  -  0.  I .  •  ■  ■  .  (r  -  1  i  and 

ir  r  v.  There  are  exactly  (r  -  2)  distinct  values  of  n  and 
these  define  the  following  ir  -  2)  paths: 

( .»  .  V  I  —i >  (A  .  tl  I  — *  I  If.  v )  — *  I »  .  i )  — >  (  v.  it  I  — »  (  t .  v ) 

These  ir  -  li  paths  shown  above  are  all  disjoint  Paths  2 

through  ir  -  2)  are  disjoint  because  ot  the  value  ot  a  .  distinct 


for  each  path 

Path  1  is  disjoint  with  paths  2  through  ir  -  I )  since  »  *  a 
and  tc  *=  y. 

Thus,  it  remains  to  be  shown  that  there  exists  one  addi¬ 
tional  path  from  xk  to  xk  that  is  disjoint  from  the  above 
(r  -  I )  paths.  The  following  constructs  such  a  path,  denoted 
as  path-/-  for  r  =  even  and  r  =  odd.  separately  . 

r  =  even:  First,  it  may  be  noted  that  as  per  the  con¬ 
struction  procedure,  nodes  xk  and  xk  are  directly  connected 
to  some  nodes  uk  and  vk  (in  Rj.  respectively 

i)  Let  u  =  y  (thus,  v  =  a).  Flence.  xk  and  xk  are  con¬ 
nected  by  the  link  (xk.  xk ).  Path  r:  xk  — »  xk. 

ii)  Let  u  *=  y  (thus,  v  =7  a).  Hence,  xk  and  xk  are 
not  connected .  Path  r:  ( x  .  x  )  — *  («,«)—»  (  u  .  u )  — * 
(i/.  v )  — »  (  y.  y). 

This  above  path  is  disjoint  from  the  paths  1  through  (/•  -  I ) 
shown  above  since  a  *  it  and  v  P  v. 
r  =  odd:  path  r:  xk  — *  n  — »  xk. 

This  path  passes  through  node  n  which  has  not  been  used 
in  any  of  the  above  paths. 

Since  FGir.  2)  networks  are  of  degree  r.  the  following 
theorem  is  a  direct  consequence  of  the  above  discussions. 

Theorem  7:  FGir. 2)  networks  arc  optimally  fault  tolerant, 
and  failure  of  any  (r  -  i  I  components  is  tolerated. 

Example  2:  Consider  the  network  shown  in  Fig  2  Here 
/•  =  4  Hence,  the  network  is  3-fault-tolerant,  l.et  the  nodes 
0.  5.  and  7  be  faulty. 

Given  these  nodes  as  faulty,  one  can  construct  the  follow¬ 
ing  path  from  any  s  to  d.  which  will  be  fault  free 

<.v,..v„)  — ►  ( V , .  2 1  — >  (2.5,1  —  (2.2) 

—  I  2.  (/;  i  —  id  .2)  —  (d, .dp 

Thus,  given  \  -  4  and  d  13.  one  has  the  following 
path: 

4  h  —  q  —  It)  —  1 1  —  14  — *  13 

Fault  Tolerance  <>t  FGir.  mi  Networks,  m  *  3  The  fol- 
lowing  definitions  and  lemmas  will  be  useful  in  constructing 
the  paths 

Definition  Let  wt  1 1 1  represent  the  number  of  occurrences 
of  i  in  radix-/-  representation  of  i 

For  example,  if  i  -~  (0.  1 .  1 .  2.  1 1  in  radix-/-,  u  /,,1 1 1  I . 
wt i ( a  )  -  3.  u7 -(  a  1  ^  I.  and  wt  ,1  a  I  0 

Lemma  3  II  ivMaI  *  wt  t  \  ).  I'oro/ty /.  0  ■  /  •  (/  I  i. 
then  i  *-  xx. 

Let  ptii.j)  represent  the  path  from  /  to  /.  as  defined  m 
Section  II 

The  path  pti  i.  /)  denotes  a  path  ot  length  at  most  (2 m  I  i 

lemma  4:  Given  any  node  i.  \  *-  s.  \npti\.ik  i.  ik  /  R, . 
the  following  relationship  holds: 

Wt  111  •  U7  111  if  1.,  *  I 

Wt  111  ■  11  /  (  1  I  ill,  I 

II  t  I  1  I  •  U 7  III  It  1*1 

l.emma  3.  Two  paths  ptis.ik  I  and  ptix./k  I  arc  node  dis¬ 
joint  if  i  >  / 

Proop  l.et  i  /  pti  i.  ik  i  and  i  e  pti  i.  /A  i  denote  any  two 
nodes  in  palhs  pti  s .  ik)  and  pti  s .  jk  i.  respectnely 
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Case  I:  Let  s0  =  i. 

Thus,  one  has.  from  Lemma  4,  wr,(  x)  >  wr,(r),  but 
wtj(x)  s  wtj(s) .  Hence,  x  *=  v. 

Case  II:  Let  s0  =  j. 

Similar  to  Case  I. 

Case  III:  Let  j()  *  i  and  st)  *■  j.  Here.  vtf,(.r)  >  wt,(s), 
and  wt,(  v)  <  u7,(s).  Thus,  wt,(x)  =*  wt,(  y),  and  hence 
x  ^  v.  Q.E.D. 

The  following  Lemmas  are  direct  consequences  of  the 
above  Lemma. 

Lemma  6:  Given  any  (r  -  1 )  faulty  nodes,  there  exists  at 
least  one  node  xk  where  xk  e  Rk,  such  that  the  path  pt(s.xk) 
is  fault  free. 

Lemma  7:  Given  any  (r  -  1 )  faulty  nodes,  there  exists  at 
least  one  node  xk  where  uk  e  Rk,  such  that  the  path  pt(  yk.d) 
is  fault  free. 

The  following  shows  it  is  possible  to  travel  from  any  s  to 
any  d.  in  spite  of  faults  in  any  (r  -  1)  nodes.  This  is  illus¬ 
trated  by  considering  the  following  cases  separately. 

Case  I:  All  of  the  faulty  nodes  belong  to  the  subset  Rt. 

The  subset  /?*  contains  r  nodes.  Since  the  number  of  faulty 
nodes  does  not  exceed  ( r  -  1).  there  must  be  at  least  one 
node  xk  which  is  fault  free. 

Consider  the  path  pt(s,xk)  This  entire  path  must  be  fault 
free  since  all  of  the  intermediate  nodes  in  the  path  pass 
through  nodes  in  ,V  -  Rk 

Similarly,  the  path  ptlxk,  d)  must  also  be  fault  free.  Hence, 
the  following  path  from  s  to  d  must  be  fault  free: 

s  — *  ■  ■  •  pH  s.  xk  )  ■  •  •  — *  xk  —i *  •  •  •  ptLx.  kd  )•••—*  d  . 

The  length  of  the  above  path  is,  at  most,  equal  to 
(2m  -  1)  t  (2m  -  I)  =  (4m  -  2). 

Case  11  All  of  the  nodes  in  the  subset  Rk  are  fault  free. 

In  this  case,  the  faulty  nodes  are  thus  confined  to  the  subset 
N  -  Rk. 

As  per  Lemmas  4  and  5.  one  always  has  two  fault-free 
paths  of  the  type  ptis.  xk )  and  pH  xk,  d) 

The  following  constructs  a  fault-free  path  from  s  to  d. 

If  x  =  v.  then  follow  the  path  given  in  Case  I:  else,  if 


X 

>  V, 

and 

X  - 

v  s  r/2  or 

X 

<  v. 

and 

V  — 

x  >  r/2  then: 

follow  the  path  giver 

i  below 

in  ( 

1 );  otherwise,  follow 

given  in  (2) 

s  — *  pHs. 

xk )  ■  ■ 

■  —  xk 

— ► 

l.r  -  !)*-»••■ 

-*  (  V 

*  1  )k 

— *  xk 

-  • 

•  •  pH  xk .  d  )•••—»  a 

v  — »  pH  v . 

xk )  •  ■ 

■  —  xk 

- 

U  +  Ilk  —>■■  ■ 

—  (  V 

-  II  k 

-*  xk  ■ 

-  • 

■  ■  pH  xk.  d  d 

Here  the  maximum  path  length  is  equal  to 


(4m  -  2)  +  r/2  , 

Case  III  Nodes  in  both  F,  as  well  as  in  ,V  -  Rk  are  faulty. 
As  before,  there  always  exist  fault-free  paths  pHs.xk)  and 
pH  xk.  d). 

The  following  shows  that  there  exist  at  least  r  disjoint 
paths  between  any  xk  and  \k.  Consider  the  following  paths 


path- 1 :  xk  — »  (x  +  1  )(k  — *  •••—»(  y  -  1  )*  — ►  xk 
path-2:  xk  — *  U  -  1  )k  -*•-■—»(  y  +  1  )k  —*  xk 
path-3  through  r.  for  all  w.  0  <  w\  <  (r  -  1)  and 

w  #  x  and  w  £  y . 

consider  the  following  (r  •  2)  paths: 


xk 

=  (x.  ■  ■ 

. .  x,  r) 

Lx.  ■  ■ 

. ..t.w) 

Lx.  ■  ■ 

■  ■  ■  ,  x.  ss.x) 

u.  •  ■ 

■  •  •  ,x.  w.  y) 

Lx.  ■  • 

•  ,.r.  H.y.  r) 

(x,  ■  • 

•  ,  .r,  H’.y.y) 

( u\  V, 

y. . ,y) 

( y,  y. 

. .  y.  w) 

xk 

=  (  V.  V. 

. .  v> 

Since  w  *=  .r  and  »  t  v,  there  are  exactly  (r  -  2)  distinct 
values  of  w.  Each  one  of  these  distinct  values  defines  a  path 
and  in  this  set  all  of  the  ( r  -  2)  paths  are  disjoint  since  the 
value  of  w  for  each  path  is  different.  Now  it  may  be  seen  all 
of  the  r  paths  shown  above  are  disjoint.  The  intermediate 
nodes  in  path- 1  and  path-2  have  only  one  digit  in  their  repre¬ 
sentation.  whereas  the  intermediate  nodes  paths  3-r  have  at 
least  two  digits  in  their  representation  Therefore,  there  can¬ 
not  be  any  intermediate  nodes  that  are  common  to  any  two  of 
the  above  r  paths. 

The  maximum  path  length  here  is  given  as 
(2m  -  1)  +  (2m  -  1)  +  (2m  -  I)  =  (6m  -  3). 

The  following  theorem  is  based  on  the  above  observation. 

TheoremH:  Given  any  FG(r,m)  network,  there  exists  a 
path  from  any  node  to  any  other  node  in  spite  of  (r  -  1) 
faults  of  length  at  most  (6m  -  3)  and  this  path  can  be  con¬ 
structed  algorithmically. 

Example  J:  Consider  the  FG(4.3)  network  which  has 
64  nodes  and  is  of  degree  5. 

Let  s  =  5  and  d  =  22. 

Here.  Rk  =  {0,21.42.63}.  The  following  illustrates 
Case  I  and  Case  III 

Case  1:  Let  nodes  0,  21.  and  42  be  faulty.  Using  the  tech- 
nique  shown  in  Case  1,  one  constructs  the  follow¬ 
ing  path: 

5  — *  ■  ■  •  pf(5. 63) - *  63  —  •  pH 63.  22) - *  22 

which  is  equal  to 

5  — >  7  — *  28  — *  31  — *  6 1  — *  63  — >  61  — *  55  — *  23  — *  22  . 

Case  III  Let  nodes  0.  21,  and  23  be  faulty.  Using 
Case  III.  one  has 

5  —  ■  pH  5.631  •  •  ■  —  63  —  42 

—  •  •  •  pt(42,  22) - >  22 

which  is  equal  to 
5  — *  7  — *  28  — »  31  — »  61  — *  63 

— *  42  — »  4 1  — *  38  — »  37  — *  22  . 
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Fig  7  Nodes  in  £3 


i  - 1 


Fig  X  Detour  method  of  analyzing  connectivity 


In  the  above,  the  fault  tolerance  of  FGir.  m I  networks  has 
been  shown  to  be  at  least  equal  to  ( r  -  I).  By  using  an 
analysis  similar  to  that  given  for  FGi2.rn)  networks,  it  can 
also  be  shown  that  the  fault  tolerance  of  FG(r.m)  networks 
for  m  -  even  is  equal  to  r.  and  thus  is  optimally  fault  toler¬ 
ant  However,  it  remains  to  be  seen  whether  FGir.tn)  net¬ 
works  for  m  =  odd  are  also  optimally  fault  tolerant. 

V.  Conch 'sion 

This  paper  presents  certain  regular  networks  with  optimal 
I  near-optimal  I  fault  tolerance.  Various  fault-tolerant  proper¬ 
ties  of  these  networks  are  summarized  in  Table  II  Further 
research*  of  interest  here  is  the  VLSI  layout  |8|.  as  well  as 
internal  testing  and  self-diagnosis  [l].(4|-[6|  of  these  net¬ 
works  The  binary  FG (2,  m)  network  can  be  considered  as  a 
supergraph  of  the  shuttle-exchange  graphs  |K],  this  is  known 
to  possess  efficient  VLSI  layouts  |9|.  1 1 H | .  Also  of  interest  is 
how  best  one  can  utilize  the  large  degree  of  fault  tolerance 
available  in  these  networks  for  both  yield  enhancement  as 
well  as  for  fault  tolerance. 

It  may  also  be  noted  that  the  graphs  presented  here  ma\  be 
of  interest  in  the  context  of  id.  k  )  graphs  |4|.  Specifically,  the 
FG ( 2.  rn)  graphs  with  even  m  and  FGir.2  I  graphs  provide 
optimum  connectivity  and  small  diameter  for  both  the  origi¬ 
nal  graph  and  the  subgraph  obtained  after  deleting  faulty 
nodes  1 19|.  |20|.  Other  recent  work  in  the  area  1 2 1 1  -|21 1  also 
have  addressed  these  problems 

Finally,  it  may  be  added  that  the  detour  technique  provides 
a  tool  lor  analyzing  the  connectivity  |24|  of  arbitrary  graph 
One  way  to  establish  that  the  connectivity  ol  a  given  graph  is 
i  is  to  prove  that  there  exists  at  least  <  dis|oint  paths  between 
every  pair  ol  nodes  But  this  requires  a  computation  of  Oin  i 
for  an  n  node  graph  However,  an  alternate  approach  would 


be  to  establish  that  there  exist  at  least  (<  -  I )  independent 
detours  around  every  node  (connecting  every  pair  of  neigh¬ 
bors)  as  illustrated  in  Fig.  8.  Such  an  analysis  may  be  simpler 
at  it  would  require  a  computational  complexity  of  Oin)  for 
bounded  degree  graphs. 


Appfndix  A 

Let  Q  represent  the  set  of  four  nodes  <0 .k.q.k  -  q). 
These  four  nodes  arc  interconnected,  as  shown  in  Fig.  7 
Let  ,V  -  Q  represent  the  set  of  (n  -  4)  nodes  which  con¬ 
sist  of  all  nodes  other  than  the  nodes  in  Q 

To  begin  with,  it  may  be  seen  that  given  any  node  s  one 
can  construct,  from  v.  three  disjoint  paths  to  nodes  0.  k.  and 
q.  below. 


KI.O.  ■  .  (Mil  (  I .  I  .  •  .  I  i  HI.  | .  ().  1 .  •  ■  .  0.  I  i 

o  k  q 

V.  I  m  even 
other  c ases  are  similar 


revilin'.  I  VI  I  l-lol  I  K  \\l  M  I  vv,  >Rk  VRl  IIIII  CTI  HI  s 


In  constructing  these  paths.  it  is  assumed  that  m  -  even  and 
\  =  v  ,  I .  (Proof  of  disjointedness  can  he  readily  derived. ) 
Similar  paths  can  be  constructed  for  other  combinations  of 
(  v  .  vn)  as  well  as  when  m  -  odd  Furthermore,  it  can  be  seen 
that  there  exist  three  paths  from  0.  k.  and  q  to  any  destination 
node  </  where  the  paths  are  disjoint  The  length  of  these 
disjoint  paths  can  be  seen  to  be  at  most  equal  to  2m.  These 
observations  form  the  basis  for  the  following  path  construc¬ 
tion  procedure,  which  constructs  paths  from  s  tot/,  given  two 
faulty  nodes  in  the  network.  This  is  described  by  considering 
various  cases  which  correspond  to  different  distributions  of 
faulty  nodes  between  Q  and  ,V  -  Q. 

Case  I:  Both  the  faulty  nodes  are  in  ,V  -  Q. 

This  implies  that  all  the  four  nodes  in  Q  are  fault  free. 
Since  there  are  three  disjoint  paths  from  v  to  nodes  in  Q.  one 
can  reach  at  least  one  of  the  nodes  in  Q  w  ith  two  faulty  nodes 
in  ,V  -  Q.  Similarly,  one  can  reach  d  from  one  of  the  nodes 
in  Q.  Since  all  of  the  nodes  in  Q  are  fault  free,  one  can  reach 
</  from  v  through  Q  by  using  a  path  of  length  at  most 
2m  -  2  *  2m  =  4 m  -  2  | there  exists  a  path  of  length  2 
between  any  pair  of  nodes  (Fig.  7)]. 

Case  II :  Both  the  faulty  nodes  are  in  Q 

In  this  case,  there  must  exist  at  least  one  node:  0.  q,  or  k. 
through  which  a  path  can  be  constructed  from  v  to  d. 
For  example,  assume  nodes  0  and  q  are  faulty.  Since  the 
paths  from  v  to  0.  v  to  q.  and  v  to  ilk  are  disjoint,  one  can 
reach  k  in  spite  of  these  two  faults.  Similarly,  from  k  one  can 
reach  d  The  total  path  length  here  will  he  equal  to  Am. 

Cau  III  Faulty  nodes  in  both  Q  and  V  -  Q 

In  this  case,  there  is  a  node  in  Q  and  a  node  in  V  -  Q  .  both 
ol  which  are  faulty,  thus,  the  three  remaining  nodes  in  (J  are 
tault  free  and  are  connected.  W  ith  a  single  faulty  node  in 
V  <J.  one  can  reach  at  least  two  of  the  three  remaining 
fault-free  nodes.  Similarly,  d  can  be  reached  from  at  least 
two  of  these  three  fault- free  nodes  Thus,  there  must  exist  one 
node  ii.  k.  or  q  through  which  an  entire  fault-tree  path  from 
v  to  d  can  be  constructed.  Thus,  this  path  is  of  leneth  at  most 
4m  -  2 

AlT!  M)IX  B 

Cl  INS  I  kl  Cl  ION  Ol  1)1  1 1  M  KS 

Can  Kl  Here,  the  original  path  contains  the  lollowmg 
sequence  ol  nodes  represented  in  radix-r  as 

ii  i  n .  . . ,n.  ii  I 

faulty  v  i  u..  . . .  ii  .  i  I  i  *  ii. 

u  in..  .  •  .  ii  .<  .  ii.  I 

First,  it  will  he  shown  that  there  exists  a  limit  tree  path 
Irom  ii.  ot  the  type  shown  below,  for  some  i.  Then,  this  path 
will  he  used  as  part  ot  the  detour  from  u  to  tv 

ii  Hi..  . . .n.u.  I 

'u.,  . . .  n  .  \  i 

in,.  -.  ■■■  .u  .  \.  ii..  i  . 

In  order  to  prove  this,  we  w  ill  consider  two  separate  eases 
I  irst.  we  consider  the  case  r  V  anil  next  consider  >  4  4 


4,< 

i)  Let  r  =  i:  Thus,  one  has  t  =  I:  therefore,  there  is 
only  one  faulty  node  i>.  Consider  the  path 

u  — *  (u, „  [.■■■.  lit.  e)  —*  (u,„  ;.•••,  u ,.  e.  u, „  ,) . 

This  path  must  be  fault  free  because  the  node  (u„ 

ut.  e)  cannot  be  the  faulty  node  v  since  e  4=  r.  the  least 

significant  digit  of  v.  The  other  node  (u . •  •  •  .  u,.c.  u„  ,) 

also  cannot  be  faulty  because  then  one  has 

(!/,„.  :.•••.(/,.  e.  u,„  i)  =  v  =  (a,„  <•) . 

This  would  ir.  turn  imply  e  =  e.  a  contradiction  of  the 
definition  of  e. 

ii)  Let  r  >  4:  Consider  the  following  two  subcases, 
the  first  corresponding  to  all  of  the  t  faulty  nodes  that  have 
the  same  digit  in  the  least  significant  position:  that  is. 

=  •  •  •  =  fa  =  e.  the  least  significant  digit  of  v.  The 
second  case  corresponds  to  when  the  above  is  not  the  case. 

a)  Let  /,',  =  =  •  •  •  =  fa.  Consider  all  the  /i-neighbors 

of  ii.  There  are  altogether  (r  -  I)  of  these.  No  two  of  these 
nodes  have  the  same  digit  in  the  least  significant  position. 
Since  all  of  the  faul'y  nodes  have  the  same  digit  in  the  least 
significant  position,  only  one  of  these  /(-neighbors  of  u  can 
be  faulty  and  this  node  is  v.  Thus,  the  remaining  (r  -  2) 
/(-neighbors  must  be  fault  free 

Without  loss  of  generality  ,  let  us  assume  these  nodes  to  be 

III,,,  .•••.!/,.  Iltm  •  4  .  «|.  I  I.  •  •  •  . 

(i i.  •  •  •  .  ii  ./•  -  4 1  in  radix  r  . 

Consider  the  following  ( r  -  2)  paths  from  u  to  the 
e -neighbors  ol  these  ir  -  2)  nodes  shown  above: 

Fault  tree  At  most  t  arc  faulty 


‘  U  .  ■ 

. .  ii  .0 1 

- ♦  U4., 

.i/,.().  ii„  i 

(//...  .  • 

-  — *  in,..  .. 

.  «|.  1 .  u„  ,  I 

'in..  . 

•  4  .  n  .  r  -  }  1 

- *  1  H...  ' 

•  .  U.  . 

r  -  }.ii.„  ,) 

flic  ii  2i  nodes  shown  m  the  right-hand  side  are  seen  to 
be  all  distinct  l  herelore.  at  least  it  -  2  i)  of  these  are 
tault  free  Fort4  44  4  .  one  has  it  2  M  -  I  There  must 
be  at  least  one  path  ol  the  type  n  — •  i n„  \  1  -— 

i ■  ■  4  .  n  .  « .  ii  ..  I  that  is  tault  tree 

For  i  T  one  hast  I .  there  lore .  there  is  only  one  node 

that  is  taulty .  the  node  r 

Now.  consider  the  second  ease  lor  which  all  ol  the  /  digits 
appearing  in  the  least-significant  position  ol  the  faulty  nodes 
are  not  distinct  Thus,  at  most  (/  I  I  distinct  digits  appear 
in  the  least-significant  position  Consider  the  (r  Ii 
Ii  neighbors  ot  u  All  ot  these  nodes  have  distinct  digits  m 
the  least-signitieant  position  So.  at  most  it  2i  ol  these 
/i-nvighhors  can  be  faulty. 

Consequently,  there  must  be  at  least  (/  •  I)  of  these 

Ii  neighbors  that  must  be  fault  free  since  (r  -  I)  -  2t  As 

hetore.  without  loss  ol  generality,  we  can  assume  these 


_■  v  . 


& 

v>’3 


% 


U  +  1 )  neighbors  as  (u„  u ( ,  0 ) .  u,.  1 ). 

M| ,/ )  in  radix-r.  Consider  the  following 
(/  +  l)  g-neighbors  of  these  nodes,  through  these  nodes: 


Fault  free 


At  most  /  are  faulty 


(Um  1-  ■ 

■  •  ,  u, .  0) - * 

(a,*.:.-- 

•  .  M|.0.  um 

(u„  ,.  ■ 

•  •  ,  U 1 .  1 )  - * 

(Um-2.  •  • 

•  .  u,.  1 .  u„ 

^1//-  1.  • 

•  •  .  It, .  1 ) - * 

<!/„  :•  ■  ■ 

■  ,u,.t.u„ 

Since  the  (t  +  1 )  nodes  in  the  right-hand  side  are  all  dis¬ 
tinct.  there  can  be  at  most  /  of  these  that  can  be  faulty.  Thus, 
there  must  exist  at  least  one  path  of  the  type  u  — *  -  . 

u,.x)  — *  ( um  •  ,  M|jc,  um  i)  that  is  fault  free. 

Now  consider  the  following  path  from  u  to  vt  that  uses  the 
fault-free  path,  as  constructed  above. 

b)  Let  *  f{,  for  some  i  *  j,  I  s.  i,  j  s  t.  Here,  con¬ 
sider  the  following  two  separate  subcases.  First  assume  that 
the  /-digits  fL  •  •  ■  .f„  appearing  in  the  least  significant 
position  of  the  faulty  nodes  are  all  distinct. 

One  always  has  at  least  (r  -  1  -  /)  of  the //-neighbors  of 
u  which  are  fault  free.  Let  these  fault-free  nodes  be  repre¬ 
sented  as 

(//„,  !.•••.  M,.0).  Ut„  I  ).••■  . 

( um  ,.•••.  it,.  r  -  2  -  /) . 

Now  consider  the  following  g-neighbors  of  these 
(r  -  I  -  /  )  nodes.  These  may  be  represented  as 


.  U |.().  ll„  I  ).  (//,„  '.•••.  Ml.  I  .  U„  |). 


f.  u„  1  I  . 


Note  that  these  nodes  are  all  distinct  and  have  the  same  digit 
in  the  least-significant  position.  According  to  the  hypothesis, 
no  two  faulty  nodes  have  the  same  digit  in  the  least- 
significSnt  position.  Thus,  only  one  of  the  above  nodes  can 
be  faulty  the  remaining  (r  -  2  -  t)  of  these  must  be  fault 
free.  Forr  2  4.  one  has  tr  -  2  -  /  )  2  I  Consequently,  in 
this  case  one  has  at  least  one  path  of  the  type  shown  below 
that  is  fault  free 

it  — *  ( tt„.  it, .  v )  —*  ( //,„  ;.••■.//..  .v.  It..,  |  > 

it  ~  lit,,,  ,. . .  iti.it,, >  fault-free  segment 


It  — *  ( It,,. 

,.■■■  .It  |..V 

lit,,,  ,.  — 

•  •  •  .  It,.  It,.) 

1  It.,,  , .  — 

—  .//,.<) 

III...  ;.  ■  ■  ■ 

.lt,,X.lt„.  i) 

iu„, 

■  •  .  It  1 .  .1 .  ( ■) 

i  C  .  l(„}  : ' 

■■■■.  It  1 .  \ ) 

(  (’ .  Uir  ;  .  '  ' 

•  •  •  -  .  M 1  .  t  ■  1 

(m,.  >  ■  ■  ■ 

■  ■  .  It  1. 1  .  f) 

;.  •  *  * 

.U,.f.lt„.  |l 

.(instructed  as  above 


most-sigmlieant  position; 
hence,  the  nodes  are  fault 
free 


Case  R2  it  — *  t'  g(/<)  — *  11 
to  the  following  path  segment: 


/id1).  This  corresponds 
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U  =  {U„.  ,. . .//].«„) 

faulty:  t»  =  (um  ;.  •  •  ■  um  ,) 

**'  =  (««,:. . •  M|.  (')  c  *=  u„  i  . 

Now.  consider  the  (r  —  I )  //-neighbors  of  u.  All  of  these 
are  distinct,  and  none  of  these  is  the  faulty  node  v  since  this 
node  cannot  be  both  g  and  //-neighbor  simultaneously.  Be¬ 
side  v.  there  are  (/  -  1)  faulty  nodes;  thus,  at  most  (/  -  I) 
of  the  //-neighbors  of  u  can  be  faulty.  This  therefore  implies 
that  there  are  at  least  (/  +  l)  //-neighbors  of  u  that  are  fault 
free  since  r  —  1  >  2t.  So,  using  arguments  similar  to  those 
given  in  Case  R1  (when  the  least  significant  digits  of  faulty 
nodes  are  not  distinct),  one  can  assert  that  there  exists  a 
fault-free  path  of  the  type 


u  — »  («„. 


,u ,,jr)  — »  (u„-2.  ’  ■  ■  .  w ! . x,  um~ 


Now,  consider  the  following  path  from  u  to  v  that  uses  the 
above-described  fault-free  path; 


«  =  (wm-i.  •  • 

■  ■ 

(tin,-:.  '  '  ' 

(u„  ;.  ■  ■ 

( e .  //,„.  ;. 
(<’.  ■ 

>V  =  (/<„-:.  •  •  ' 


■  '  ’  .  Mi .  «o) 
■■■■  .14,,  X) 

.  U,.X.  U„- ,) 

•  •  .  Mj,  x.  e) 

■  ■  •  •  .  M|..t) 

’  ’  ’  ’  .  M| ,  Mu) 

•  ■  .  M|,Mn.e) 
••  .  M, .  M„.  C)  . 


nodes  cannot  be  faulty,  as 
e  appears  in  the  least-  or 
most-significant  position. 


Case  R3:  u  —■ >  v  =  g(u)  — >  w  =  g(v).  This  corresponds 
to  the  following  segment: 


it  =  (w*-i. 
V  =  (//,„-;. 


. .  U,.Uo) 

.  U„-,  I 


»■  =  I.I4,„  :) . 

As  in  Case  R2.  one  can  construct  a  fault-free  path  of 
the  type 

it  — >  (//,„.,,  •  •  •  .«,..<■)  -»  ■  ■  •  .u,.x.  u„.,) . 

Consider  the  ( r  -  1)  //-neighbors  of  the  node 
(i/,„  ■  .it, .x. None  of  these  nodes  has  u„. ,  in  the 

least-significant  position;  hence,  none  of  these  is  the  faulty 
node  v.  Since  there  are  (/  -  1 )  faulty  nodes  besides  v.  at  most 
u  -  I )  of  these  //-neighbors  can  be  faulty.  Thus,  there  must 
be  at  leuNt  (/  I)  of  these  //-neighbors  that  are  fault 
free.  Without  loss  of  generality,  these  nodes  can  be  assumed 
to  be  (  It.,,  U  I  .  V  .  0  )  .  (  U,„  ;.■■•.«  I  .  .V  .  1  )  .  '  •  •  . 

(//,..  it,,  x.  t).  Consider  the  g-neighbors  of  these  nodes, 

as  shown  below  : 

III.,,  It,  .  X.  0.  Il,„  ;).(«„ ,  It,.  \.  I  ,  It, „  -I. 

■  •  ■  .  lit...  1.  /.  It..  ;•  . 

These  (/  *  I)  nodes  arc  all  distinct,  hence,  at  least  one  of 
these  must  be  fault  free  Let  this  lault-frce  node  be  (//„,  i. 
••'.//,.  i.  v.  /(,.,  )  Thus,  a  fault-free  path  ol  the  t\pe  shown 


r-VsV- 


ww  urns mmm 
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below  is  easily  constructed: 

II  — *  [((„,  H|.  t)  — » 

—>  (u„  W|.  .V.  (/„,  I  )  — *  III,  ;.•••.  U.X.X ) 

— »  (H,„  !.•••.  M |,.V.  V.  (/„  4  . 

Now.  using  this  path,  one  can  construct  a  detour  from  it  to 
w,  as  illustrated  in  the  following: 

U  =  (l/,„  . .  M|.  U(i) 

(«,„  |. . .  M i .  -V ) 

(i<„,  ;. . .  Ml,  A.  |  ) 

(w,„  ;. . .  M|..v.y) 

(M,„  ).••••.  It | ,  -V .  V.  U„  I  ) 

ln,„  ,. . .it i.Jt.v. «*) 

U\it,„  i. . .  i<|..V.  V) 

|C.M,„  >. . .  U|..t.  <*) 

ie.e.it,„  >. . . «|..v)  e  appears  in  the  most-  or 

l  . . . least-significant 

position;  hence,  these 
'• . •  e *  nodes  cannot  be  faulty 

U\  . .ltn.lt.,,  |> 

ln,„  :. . .  it,,,  it. „  i.  e) 

Vt  =  III.,,  lln.  U,„  I.  lt,„  4  . 
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summarizes  our  results. 
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1 :  In  .n  isynor.-onous  env  i'-onm-rnt , 

i  fault  :s  ,;y  a- mt  :  to  :  :  :•  o  o  v  e  r  e  in  th* 
;  r-j-r.ee  f  it  t  f-ult.s  I  ff  .  •  til. 

Proof :  In  an  asy.  "'.r  .sjja  environri'  n-,  t  n. 

:  s  r  ave’-er  of  a  fill'  -.avisos  the  rows  r«  fcj 
or  >.»  j  ,'t sting  me  iif  nitirn.  'lodes  in  the  network 
•eoeiving  a  Inf  art  i  m  ,  :  ■■  t  the  diagnosis  iff 
tn.e.,  "v.vii.  It  f-'y:  i  tn-‘y  jubse-i  ient  !y  test 

•  i  Tms  t  ah  i-oro  i:  ■  as  t  strategy 
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III.  ASYNCHRONOUS  [£0_ P  DIAGNOSIS 
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r**  ;  ,  i  r*  r-  n'  a  f..r  .  H."iif  r.:ar,  era;-*.  '  '  h**  *  i;  ahl** 

yr  i  o  .  ri  . "  i*  .*.*.  .i,  'ir'o-iit.  Gr.'apias  'trust  .Iso 
!*•■  -.■■ih.,:  5**r**  i  '  n  .'  .  r*_*  ret  H  u  i  ;  t  on  i  an  'r  to*;  one 
r.-u  .*' ;  i  '  •  ;  .  n  •  •  f  *',*ur.  r,**ot  i  Oil 

w i  a  i  f  i  •*  ‘ '  :  r  ‘  i  *  *  1  .■  *.'•'.••  'hi'  t  '*?•■;'  t  s  as 


input  i  graph ,  3,  an  :  i  1  oop,  u,  an.l  constructs  » 
testing  graph  with  only  '.'no  1  top  .  L )  .  Then  it  is 
s!io*t,  that  the  testing  graph  s'  'unst-uet-.i  will 
discover  at  least  one  fault  i  f !'  i  graph  forced  only 
of  L  can  discover  a  fault.  Then  follows  .« 


loop  in  the  presence  of  faults. 
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Let  G  =  (V, E)  De  a  graph  or.  n  vertices  witr.  a 
subgraph,  L,  consisting  of  <.  vertices  from  a 
circuit  of  0  and  the  edges  comprising  that  .'ircuit. 
An  algorithm  is  presented  that  produces  i  •  est  ing 
graph,  T  »  i V *  , E ’  ) ,  with  the  minimal  n  edges,  and 
which  includes  only  one  circuit,  L. 

Algori  tnm  TrGKMER  (  1 ,  i. ) : 

(1)  Let  V  -  V.  Let  E'  -  .-.1  ilrected 

edges  in  one  direction  of  the  circuit  L  ■ .  Let  N  « 
•  v  0  y  :  v  ( a  0)  0  E'  0  *  v  }  •  i  ,e. ,  N  is  tile  set 
of  all  nodes  in  V  not  tested  according  to  E' . 

(2i  V’-H  is  the  set  of  nodes  in  V  not 
included  in  N.  Let  ’{•  =  :  v  ■  N  :  -}  a  0  •/* -W  such 

tnat  Ca.v)  6  E  J  and  let  a(v)  be  i  function  'i  v  « 

N '  returning  an  arbitrary  node,  a,  for  whi  in,  a.v) 
6  E.  a(v)  will  oe  the  tester  of  v. 

(  3  )  Let  E  *  *-  E '  J  i  ( 5  >.  v )  ,  v )  :  v  0  N '  : . 

He  new  ;;  .  i  v  <?  V  :  V  (a.bl  0  E'  o  *  v  i. 


u )  If  N  <  j  then  go  to  v. 


;y,  E  '  )  , 


It  is  obvious  that  the  only  circuit  in  t r,- 
testing  graph  thus  formed  is  the  or.--  use  t  t.s  a 
cast 3  for  its  construction.  The  algorithm  will 
terminate  unless  ‘I  *  .  j  uni  le  1,  «  This, 

however,  cannot  occur,  because  it  would  ju i 
that  no  node  in  N  be  lined  to  any  .node  in  v •  - •; 

wr.ich  would  i  n  i  i  c  a  t  e  t  h  at  the  gr  a  ;  u  :  a 
discon.ne  :ted ) . 

figure  3.1  illustrates  tne  successive  s-ts 
T-:’,N,N")  in  an  application  of  71- JKVEH  to  a  '  x- 
r.d  -  around  mesh,  where  the  chosen  circuit  lonaidts 
of  tne  nodes  around  the  border  of  tne  mesh.  All 
idges  of  the  mesh  shown  are  in  E’  .  nodes  ir.  N-N’ 
are  indicated  by  "N”  in  th-ir  interior.  No  .  n 
11'  1  N  are  indicated  by  "N ' "  in  their  interior, 
flow  it  is  shown  that  TFjKMEH  yields  a  testing  c,." 
that  is  udepiute  to  diagnose  a  first  faul*  witr.  i  r.. 
flexible  testing  strategy  presentei  hero . 

Theorem  3^1  :  In  an  isyner.r  on  ti  ;  ■■nvir  i.n.m-eit 
with  test  -and  -broadc  is  t  i  i  ssom  i  r,a  t  i  on  .  f 
discovery,  TPLSMER' 3,L)  yields  i  testing  graph  tn  it 
will  diagnose  it  least  or.e  fault  in  tn-  pr-.j-n  f 
up  to  t  faults  i_ff  L  consists  of  at  least  ft 
vert i ces . 

Proof  :  With  test-ar.d-b'’ j-i  Icist  liss-minuti  n 
if  fault  discovery  is  i  r,  GM1.F  Ku  H  -  , 

ieoessary  and  sufficient  for  at  l-,st  >t  f  •  j  1  •  y 
iod-  to  be  tested  by  a  nonfaulty  nod- .  ' V  ; 

irbitrary  node  in  V.  Tons i  l«  r  its  ■in!;u-  ;  i--. 

>r  predecessors  in  the  testing  grupn.  lir.  •  n.  •••• 

ire  a  finite  number  of  nodes,  this  s-Tj-n . .  , 
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Figure  3.1:  T FORMER  Applied  to  a  End-round  Mesh 
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then 


fvc ,  so  O'  includes  it  least  two  adjacent  nodes . 
Let  a  and  b  be  any  two  adjacent  nodes  in  GL  Let  L 
consist  of  the  two-node  circuit  including  a  and  b. 
If  f  »  -l  then  L  includes  2(.  c-f )  nodes.  -.i  w 
assume  v  o-l  .  Since  Z'  is  formed  by  removing  f 
vertices  from  J,  union  nas  connect i vity ,  c,  O'  mast 
-live  - ;  r.;v- :t  i  v  it  y  at  least  o-f.  With  f  <  o-l  ar.d 
’  nneotivity  at  least  o-f,  tne  ’or.nect  i  vity  of  Z' 
: s  at  least  two. 


includes  n-f  nodes  unless  there  are  nodes  in 
.1 '  Oitsiie  L.  Let  v  be  an  arbitrary  node  in  Z '  , 
but  not  in  L.  Since  Z '  nas  connectivity  at  least 
o-f,  tn ere  are  c-f  inode-disjoint  1  paths  from  v  to 
distinct  nodes  in  L  [Deo  ?a].  Unless  L  includes  at 
least  :  i  c-f )  nodes  ,  two  of  the  nodes  in  L  that  v 
.•.meets  to  must  be  linked.  Let  these  nodes  be  y 
ir.  i  c.  Tier,  the  loop  may  be  extended  by  replacing 
f.-'  between  y  and  with  the  paths  between  v 

in  1  -  i  c h  if  y  and  z.  To  tne  number  of  nodes  in  L 
m  iy  always  u«  augmented  until  it  includes  at  least 
—  f  nodes  r-r  exhausts  the  n-f  nodes  in  O’,  r; 

iroiliry  3-?:  if  C  '  is  a  graph  formed  by 
removing  f  vertices  ihd  their  incident  edges  from 
1 ,  where  f  <  c ,  then  ¥  i  ~  V  there  exists  a 
;  i  •’  .■  u  i  t  ,  L  f  1'  on  vertices  such  that  l  i  min  ! 


■r  .of:  Ir.  tie  proof  of  Theorem  3.2,  let  i  be 
:T  The  t«.  iijacer.t  nodes  comprising  the 
1  ■ . .  *  1  a  1  t-'-hoce  circuit.  Then  i  will  be  in  tne 


jih’e  r.-f  >  c-f,  it  c-f )  >  c-f  wn»»n  f  <  c,  and 
.-■■7.  -.1  -ep.ires  i  loop  consisting  of  at  least 
’-f  ■  Theorem  3.2  guarantees  that  an 

i  -  a;  ilwny.s  -xists.  An  .algorithm  fir 

■  g  i  .  i ;  f  o  1  1  c  w  s  ■'  e  a  i  i  I  y  from  t  h  • 

■  i  of  Tieje.-m  3.  2.  lucr.  a.h  a  Igor  it  hr.  is 

•  .  .  *  i3  o  .i,r"  .j  i  r  j  j  *  i  ■  I  g  f  in  c  t  i  n  , 

:  ■  r  :  \ .  ,  t.-  .vaiintle,  »3i  m,  given  t 
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V.  CONCLUSIONS 


•or.dit  ion  will  oerUeJ  pertaining  to  the 

i:  amefer.  Let  Ki.  I'  denote  the  diameter-  of  0.  Let 
denote  the  graph  'Stained  by  removing  the  nodes 

of  r  a:.J  their  incident  edges  from  T. 


r  j*  * 
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neorern 


f  _K  >  ;o»l)  ‘  /  w  then  V  F  such 
contains  a  loop  with  nore  than 


.?e?r8bj  Cor  proof. 


n  •  the:.  it  if  2  K  >  our 

synchronous  Loop  Highest;?  procedure  may  always  be 
applied.  Vi  e  s  ;  g  g  ■■  3  t  t  h  e  r  e  are  stronger 
relationships  between 

si.:  able  loops,  L. 


K,  and  the  existence  of 


•r  research  e  juL  1  evolve  around  the 
i  r.gr  ed  i  er:t  3  i  r,  a  plan  intent  on 
! ;  igr. os ;  3  : 


V.  m  efficient  loop  finding  ilgori  tnm--even 
■he  that  -ill  hot  find  in  existihg  -adequate  loop  if 
me  time  required  would  be  exorbitant 

ar.  algorithm,  to  form  a  testing  graph  from 


v  *r.  -Tfi-'ivr.t  :-yi:  i:*t  irk»*  igree^nt  •*  lg'_>ri  tnm 
i  :  a*  t ;  :  it  ;  *;r. 

:f  1  ojp  i;5  t  >.*  <  j-iV?  t-hen 


♦  j  j  3>Ti  jr  omo -  i  nt.er  pret  i  ng  algorithm,  -is 
*v1i  r-y  tne  ;rjof  jf  Theorem  ?.2 

ol3*t* ,  tr,er  ipply  »syri-jhrom>Ks  1 3op  lit  gnosis  for 
i**4.-?:4  i  -jr\  f  tne  :wxt  ,  or,  upon  discovery  jf 

i  f  i  i 1 1 ,  1 1 -■*r t  the  system  *.  )  i p p  1  y  ".or-?  tests,  is 
•  "  :  it-?  J  r-y 


m r;  ir.  -tig  ritn.T  4  •  f  ■>r**!  •»  nonr. l f. i rr. -4 1  testing 
n  i  •:  i  i  •  r  r  e  s  p  o  n  .ling  f  -i  u  i  t  -  o  e  l  o  :  t  i  o  n 
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We  have  pursued  i  strategy  of  not  utilizing 
the  full  oapaeity  of  the  allowable  testing  graph  in 
an  effort  to  arrive  ,t  more  efficient  diagnosis. 
We  estimate  that  the  asynchronous  algorithm  of  Kuril 
ir  ’  He  Jay  iKuRe-Su]  will  prove  more  efficacious  than 
ir  synchronous  approach.  For  asynchronous 
environments,  we  have  ascertained  that  our  approach 
is  applicable  to  all  homogeneous  systems.  In 
exchange  for  not  being  -io!«  to  diagnose  tn» 
disconnecting  fault,  we  reduce  substantially  tne 
testing  overhead  required. 

We  have  not  treated  the  diagnosis  of  edge 
faults.  4s  in  our  attention  to  node  faults, 
algorithms  3FLF3  [KuRe91]  and  modified  SELF3 
iHossS?]  require  only  tnat  a  fault  be  discovered. 
It  is  manifest  that  all  edge  faults  (without  an 
incident  node  fault)  are  discovered  iff  every  edge 
in  the  communication  graph  appears  in  at  least  one 
direction  in  the  testing  graph.  The  implication 
that  at  ieust  half  the  entire  testing  graph  be  used 
may  be  unacceptable  for  some  systems.  If  a 
communication  ling  may  be  test-ad  more  facilely  than 
a  processor,  our  Ji agnostic  strategy  may  be 
•augmented  by  performing  the  additional  linx  tests 
separately . 

For  continually  on-line  systems,  testing  is 
conducted  periodically  and  faults  generally  do  not 
occur  simultaneously.  cor  those  systems,  testing 
overhead  is  reduced  essentially  by  a  factor  of  t. 
For  some  systems,  it  may  be  appropriate  to  shut 
!  own  tne  system  for  testing.  Faults  are  more 
lively  to  ippe-.tr  to  occur  simultaneously  for  such 
systems .  The  procedure  we  outline  allows  downtime 
to  be  prop  or  t  i  o.na  1  to  the  number  of  faults  that 
have  occurred  since  last  testing,  instead  of  being 
proportional  to  the  number  of  faults  tolerated. 

Not  ill  nodes  need  to  execute  LF INFER.  In 
asynchronous  onvionm*nts  the  iisooverer  of  the 
fault  discerned  by  the  syndrom-  is  Known  to  be 
r.or.f  iulty ,  so  tnat  node  can  be  tasxed  with  finding 
i  new  loop  and  the  loop  o  in  be  igreed  upon  by 
oppli'ation  if  i  Byzantine  igreem-'nt  algorithm.  In 
synchronous  •  civir  onments  the  discoverer  of  a  fault 
can  exec  ite  LFINDER  ml  broadcast  tne  loop  along 
with  news  if  the  fault.  The  connectivity  of  the 
graph  of  available  tests  guarantee.-;  that  news;  of 
every  fault  will  eventually  be  disseminated  to  il 
nonfaulty  nodes  •  Jo  inclusion  of  a  list  of  xiv-wn 
faulty  nodes  in  a  diagnostic  broad'-. a.-jt  will  allow  a 
node  receiving  new  diagnostic  information  t  -  -mew 
whether  tr.»-  node  original  iy  finding  tne  receive  i 
loop  was  yet  -aware  -if  otner  faults  in  th-  n-tw'rv. 
If  a  did:-' -pan  -y  :C-'u'-s,  then  a  r.-w  loop  c-an 


a  •.  i  ampt  1  on  tnat  -ah  las'.  all  if: 

r.  b  or  a  in  1  -mly  its.  -.--ignbors.  M<-,  .n  i  r.  i 
M  has  investigate’,  l.-'fw  -  VS  where  some  test  -- 

"ibner  m  a  /  -i  i*  be  ;  .  ,.-,,tie  j-  -»  a  y  t--  ,m;-ra--t.i  -a'.. 
Th:  a  m.ay  t,  lue  •  [a-.  ;iv-  lev  ;  ca-s ,  si'h  ,  • 

r  i  e .-,  1  hat  1  •  •  •  iv-  -  b-  aj.a  -|ty  to  I'S' 

ne ;  g'ie  ;r  i .  .  :  ,  .--I-  ;-;l-  levin's  would 
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en.rtano-1 1  wit  a  testing  capability.  A1 1  potentially 
faulty  component  •  oast  be  subjeot  to  detection. 


In  LHossoJj  necessary  conditions  ire  given  for 
a  nonnomogeneous  gmpn.  These  are  observations 
that  each  node  must  be  subject  to  test  and  I  hat 
e i c  n  none  oust  be  capable  of  discerning 

* 

interpretation  of  diagnostic  information.  Let  G  . 

* 

\V,E  i  m  1  =  vV.iC'i  with  tne  edges  corresponding  to 

» 

tne  unallowed  tests  removed;  the  edges  in  £  are 

directed . 

From  o-iT  discus's  ions  in  sections  3  and  it,  we 
Ciri  see  that  not  all  nodes  need  to  test  another 
node  to  ensure  that  a  fault  is  discovered.  Tne 
wore  of  Hosseini  determines  how  many  faults  may  be 
diagnosed  if  tne  adopted  testing  graph  includes  all 

• 

allowed  tests.  Consider  a  system,  G;G  ,  tnat  is 
it-M  ) -fault  diagnosable  according  to  [Hoss32].  We 
seek  to  minimize  the  number  of  tests  that  must  be 
conducted  periodically;  once  a  fault  has  been 
discovered,  the  dissemination  of  tne  information  is 
guaranteed  by  7hoss3?j. 


3  more  detailed  discussion  may  be  found  in 
LMeP"3oj.  For  those  no.nnomoge.neous  systems  that  we 
can  determine  to  oe  (t*l)-fauit  diagnosable  by  the 
sufficient  conditions  in  [KuReSO]  .and  in  [HossSB], 
we  :»n  employ  7FJRMER  and  an  amalgamate  of 
\JN_HoMj’  and  NON  HOMO.?  .Hoss37]  to  arrive  at  a 
mi-iimalistic  diagnisti:  strategy. 

Finally,  tne  following  suggests  a  topic  for 
f  .rtnrr  res- •  arch.  We  nave  derived  testing  graphs 
w:  ere;’;  •  :■  j  or  r.od>>  :s  tested  by  only  one  neighbor. 
This  t.  i  r.  :miz‘-s  tne  number  of  tests  required.  Not 
ill  nodes,  though,  need  test  a  neighbor.  Unless 
graph  1 3  Hamiltonian  .Deo  7-j,  some  node  must 
‘-•••st  mere  than  one  other  node.  Depending  on  the 
type  vf  testing  engaged,  it  might  or  might  not  be 
jiovenient  for  some  mod".;  to  test  many  others.  We 
r.  o  t  3 1  i  :■  1  i  s  n»i!  how  to  minimize  the  out 
:  n  •  ider.  ;e  o f  the  testing  graph,  but.  wo  conjecture 
that  t'j  tic.  do  ".e->  i  test  mire  than  two  neighbors 
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*  New  i  hallenges  have  been  brought  to  fault- tolerant  <  omputmg 
.lint  firm  essor  jri  hitec  turr  rc'searc  h  bee  Juno  ot  dcvelo/rments  in  t( 

<  tec  Irnologv  Clot'  emerging  area  is  development  of  architectures 

*  lnult  f>v  in  tori  1 1  nnec  ting  a  Ijrgt'  number  ot  proc  rssing  ('foments  on 
,i  singl r  (  hi/ 1  or  vva tor  two  important  jiojs  related  to  such  l/s/ 
processor  jrr.ns  are  the  lotus  of  this  paper.  the\  art'  fault ■  toler¬ 
ant  e  anil  w eld  improvement  techniques 

|  fault  tolerant t<  in  these  \LSI  processor  jrravs  is  of  real  practical 

|  sigoitu  ant  o  it  firos  ides  for  mut  h  needed  rehabihtv  improvement 

fheietore  we  first  itt'scribo  the  underlvmg  ronrepts  of  fault  toler¬ 
ant  e  at  worf  in  these  multiprocessor  svstems  these  pretepts  are 
useful  to  then  present  rertain  ter  hmques  that  will  incorporate  fault 
tolerant  e  mti'grallv  into  the  design  In  the  set  ond  part  of  the  paper 
wr  discuss  models  that  evaluate  how  vield  enhancement  and 
rehabihtv  improvement  mas  be  achieved  bv  certain  fault- tolerant 
tei  hmques 

^  I  Is  I K <  UH  K  1 1(  IN 

I  ho  ('volution  of  fifth-generation  computers  [44]  makes  it 
r  lejr  that  traditional  sequential  computer  architecture  will 
soon  see  a  striking  departure,  overtaken  bv  newer  architec¬ 
tures  which  use  multiple  processors  as  the  state  of  the  art 
This  particular  thrust  is  enhanced  bv  developments  in  1C 
te<  hnologv  [  30)  i  mating  a  widening  gap  between  the  tech¬ 
nologic  al  advances  and  the  arc  hitec  tural  capabilities  that 
(  an  exploit  these  tullv 

As  a  result  much  recent  research  has  focused  on  these 
new  ari  hitec  tural  innovations,  especially  those  created  by 
interconnecting  multiple  processing  elements  (PEs)  One 
important  class  of  such  architectures  is  VLSI  systems  that 
inter,  on  nee  t  a  verv  large  number  of  simple  processing  c  ells 
all  on  a  single  ,  hip  or  wafer  C  oncerns  aErouf  fault  tolerance 
m  V  l  SI  based  svstems  stem  from  the  two  key  fac  tors  ot 
n  liatulitv  and  yield  enhancements  low  yield  is  a  problem 
nl  increasing  significance  as  circuit  density  grows  One 
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solution  suggests  improvement  of  the  manufacturing  and 
testing  processes,  to  mir.mize  manufacturing  faults  How¬ 
ever,  this  approach  is  not  only  very  costly,  but  also  quite 
difficult  to  implement,  with  the  increasing  number  of  com¬ 
ponents  that  can  be  placed  on  one  chip  However,  incorpo¬ 
rating  redundancy  for  fault  tolerance  does  provide  a  verv 
practical  solution  to  the  low  yield  problem  This  has  been 
demonstrated  in  practice  for  high-density  memory  chips 
and  should  be  extended  to  other  types  of  VfSI  circuits  In 
general,  yield  may  be  enhanced  because  the  circuit  can  be 
accepted,  in  spite  of  some  manufacturing  defects,  by  means 
of  restructuring,  as  opposed  to  having  to  discard  the  faulty 
chip  Achieving  reliable  operation  also  becomes  increas¬ 
ingly  difficult  with  the  growing  number  of  interconnected 
elements  and  hence,  the  increased  likelihood  that  faults 
c  an  occ  ur 

In  the  design  of  such  fault-tolerant  systems,  a  maior 
architectural  consideration  becomes  the  system  intercon¬ 
nection  Consequently,  one  goal  of  this  work  is  the  study  of 
sound  fault-tolerant  network  architectures  that  can  be  well 
utilized  in  a  wide  range  of  VfSI-based  systems  Also,  of 
importance  are  the  related  problems  of  testing,  diagnosis, 
and  reconfiguration 

VfSI  technology  has  many  promising  applications,  includ¬ 
ing  the  design  of  special-purpose  processors  [7],  for  use  as 
an  interconnected  array  of  processing  cells  on  a  single  chip, 
as  well  as  the  design  of  supercomputers  that  use  wafer-scale 
technology  These  two  factors,  in  coniunction,  possess  the 
potential  for  ma|or  innovations  in  computer  architecture 

One  principal  aspect  of  such  architectures  is  now  fault 
tolerance  ran  well  be  incorporated  into  such  svstems  In 
c  luded  herr*  is  the  problem  of  the  placement  of  redundant 
colls  so  as  to  achieve  the  elements  of  fault  tolerance  yield 
enhancement  testability  and  reconfigurability 

II  fsilll  T<  UlRANC  r  in  VfSI  and  WSI 

Two  V  l  SI  based  areas  in  which  important  innovations  are 
l.kelv  to  or  cur  are  in  the  wafer  scale  integrated  arc  hitec 
fores  and  m  the  single-chip/multiproc  essing  element  ar 


chitoctures  The  former  has  the  potential  for  a  major 
breakthrough  with  its  ability  to  realize  a  complete  multi¬ 
processing  system  on  a  single  yvafer  This  will  eliminate  the 
expensive  steps  required  to  dice  the  water  into  individual 
chips  and  bond  their  pads  to  external  pins  In  addition 
internal  connections  between  chips  on  the  same  watei  are 
more  reliable  and  have  a  smallei  propagation  delay  than 
external  connections  The  latter  does  make  it  possible  to 
build  a  high-speed  processor  on  a  single  <  hip.  designed  by 
interconnecting  a  large  number  of  simple  Pts  these  archi¬ 
tectures  already  haye  captured  the  imagination  ot  several 
computer  inanufac turers  and  researchers  alike 

As  mentioned  earlier  the  motivation  tor  me  oipoialing 
fault  tolerance  (redundancy)  is  twofold  yield  enhancement 
and  reliability  improvement  Both  are  achieved  by  restruc 
turmg  the  links  so  as  to  isolate  the  taultv  element(s)  Vari¬ 
ous  link  technologies  arc-  available  now  which  allovy  sue  ti 
restruc  turabilitv  Inc  luded  among  these  are  tin  laser-formed 
links  \tOS  links  (tristate  logic  and  transistors)  fusible  links 
and  so  on 

Restruc  turmg  capability  is  either  static  or  dynamic  m 
tv  pc'  Which  type  is  selected  depends  on  whether  restren 
turmg  should  bo  performed  only  once  after  manutac  turmg 
or  an  unlimited  number  of  times,  as  mav  be  required 
throughout  thc>  operational  life 

file  issue  ot  fault  tolerance1  in  VISI  and  W  SI  processing 
arrays  has  bi'im  the'  sub|ec  t  of  recent  studies  eg  [,'<]  (  1 0) 
[18)  |.’0|  (.»']  |.T8),  [40]  [41]  In  these  publications  various 
sc  heme's  have  been  proposed  that  introdue  fault  tolrrane  < 
into  the  arc  hitec  tori'  of  proc  e’ssor  arrays  Bee  arise  fault  toler 
am  e  is  an  involved  sub|ect  completely  different  schemes 
might  tie  cost-effective  in  different  situations  and  tor  differ 
ent  ob|ee  tic'  tunc  turns 

Whore  evaluating  a  fault-tolerance  strategy  tor  multi 
processor  systems  we  have  to  consider  the  ■  tollowing 

asp.  i  ts 

a)  types  ot  tail  tire's  to  tie  handled  and  their  pcoliuhilitn  - 
ot  oc  c  urri'nc  e. 

ti)  the  c  osts  assoc  lated  with  failure'  oc  c  urnmc  is 

c  )  the  applti  able  recovery  methods 

d)  t)n  amount  ot  additional  hardware'  need.  ,) 

e)  the  syste  m  objective'  (unctions 


processing  even  in  the  presence  of  operational  faults,  can 
tie  t eeni’tic  lal 

tin  two  typos  ot  failures,  manutac  luring  defects  and 
operational  faults  also  ditter  in  the  costs  assoc  lated  with 
tin  Hi  Detects  are  tested  lor  bi-tore  the  I (  s  are  assembled 
into  a  system  and  therefore ,  they  contribute  only  to  the 
production  costs  ot  the  IC  s  In  contrast,  faults  occur  alter 
the  system  has  been  assembled  and  is  already-  operational 
Hence,  their  impact  is  on  the  system's  operation  and  their 
damage  might  be  substantial,  espec  ially  in  systems  used  for 
critical  real-time  applic aborts.  Clearly  a  method  which  is 
cost-effective  tor  handling  cfi'fec  ts  is  not  necessarily  cost- 
etti  c  tive  for  handling  operational  faults  and  wee  versa 
lor  both  tvjies  Ot  tailures  in  \  l  S I  a  repair  operation  is 
impossible  and  the  best  one  can  do  is  to  somehow  avoid 
the  use-  ot  the  taultv  part  bv  restructuring  the  system  This 
implies  that  in  the  water  (in  the  case  of  defects)  or  in  the 
assembled  system  (m  the  case  of  faults)  there  are  other 
operational  parts  which  are  either  identical  to  the  faulty 
one  or  that  can  tultill  the  same  tasks 

Restructuring  can  lie  static  or  dynamic  static  instructor 
mg  sc  hemes  arc'  suitable  only  to  avoid  the  use  of  parts  with 
production  flaws  Dynamic  restructuring  is  required  during 
the  normal  system  operation,  when  taultv  |iarts  have  to  be 
restruc  lured  out  of  the  system  without  human  intervention 
sue  I)  a  dynamic  strategy  might  lie  appropriate  to  handle 
detects  as  yy.  It  static  schemes  (end  to  use  c  ompaiativc  ly 
less  hardware  but  consume  o|ierator  tune  while  dynamic 
sc  hemes  arc  ,  onlrolled  intern, illy  by  the  system  and  usually 
ic  c (cure  .  vtr.i  c  in  entry 

Another  aspect  that  has  to  be  considered  when  evaluat¬ 
ing  the  eitec  live  iii  ss  nt  a  given  tault  toll  lane  c  tec  hnique  is 
th*  required  hardware  investment  III"  h. netware  added  can 
be  m  the  tnrm  ot  switching  elements  (eg  |“|  |-“j  and 
|!  |)  or  redundancy  in  processors  or  communication  links 
leg  [  1  ’|  j | )  W  lien  c  irrv  i rig  c  ml  sue  h  an  anal v  sis  u  ■  have 
!■  c  1  ak  i  u  in  i  ic  c  our  it  lie  tc  illoyy  crig  |y\  c  i  p.e  ameter  s 

)  the  relative  hardware  lompleyitv  ot  processors  com 
iimnic  ation  links  and  switching  elements  (it  they 
ex  1st  | 

I  the  sum  eptihclit'.  to  tailin'  s  (inainil  i.  tilling  dc  t*  c  Is 
*  a  operational  laults)  ni  ill  th*  itniy  me  ii'ihih  1 
'  !■  uw  nts 


tault  tolerance  strategies  can  be  designed  to  I'  at  with 
two  distent  types  ot  tailures.  namely  produi  lion  d't' el¬ 
and  operational  taults  In  the  current  technology  a  e  la 
lively  large  number  ot  defects  is  expected  when  manutac 
turmg  a  silicon  wafer  Normally  all  chips  with  production 
Haws  are  disc  arc  Jc  d  leading  to  a  low  yield  (expected  p.  i 
c  outage  ot  good  1  hips  out  ot  a  water) 

(  '(>•  rational  faults  (or  |ust  faults  |  teas  ■  ■  m  i  orup ar  w  m 
a  considerably  lower  probability  ot  occurrence  the  dd 
ferine  !■  ot  which  may  fie  m  orders  ot  mugmludi  tmproyi 
mints  in  the  solid  state  technology  and  rnatunty  ot  tie 
fabric  ation  processes  have  reduced  the  t.iiliiri  r  it.  ..I  i 
single  i  nriiponeiit  wittun  a  \  l  SI  c  tup  How  a-  i  r>e  e 
jionefitial  increase  m  the  c  omponent  count  per  \  l  si  Dnp 
ti.is  meire  than  otts.-r  the  increase  in  reliabilH'.  ot  i  s.-.gi. 
lompeme-rtt  Thus  operational  faults  cannot  t  a  gone  f  e1 
the  mgh  they  hav<’  a  substantially  lower  [trot >al  -il ,tx  ot  >  ic  c  ■.  ce 
re-nce  compared  to  production  detei  ts  (  OHS.  'Ill*  I'll  ( 
fault  toll  ram  '■  strategy  that  "nables  che  s.a.  "  t,  , 


hrcic  essmg  elements  are  traditionally  t  oiisidru  d  tin-  tiiost 
important  system  iisouih  hint'  acteeymg  1  c  |i.  n.nl 
utilization  ot  thi  in  is  many  tirru  s  uth  mjttecj  lor  exatti|tle 
in  |  w  ]  |  ■--!■]  and  |  I  j  scy  itc  lung  i  I  -  mi  nts  an  ’  add*  d  be  tyy  ■  'on 

jiroc  issciis  to.  issisf  in  ail  e  ymg  this  goal  li  i  |  ]  in  I  I  *  | 
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processors  The  effect  of  this  is  to  introduce  extra  stages  in 
the  pipeline  thus  increasing  the  latency  of  the  pipeline 
without  reducing  its  throughput 

In  the  above  mentioned  schemes,  one  of  the  underlying 
assumptions  is  that  the  extra  circuitry  (e  g  switching  ele¬ 
ments.  communication  links,  or  registers)  are  failure-free 
and  only  processors  can  fail  However,  larger  silicon  areas 
devoted  to  those  elements  increase  their  susceptibility  to 
defects  or  faults:  as  a  result,  the  above-mentioned  assump¬ 
tion  might  not  be  valid  anv  more 

In  VLSI,  the  silicon  area  deviated  to  a  system  element 
might  fie  more  important  than  its  hardware  complexity 
Consequently  100-percent  utilization  of  PEs  is  not  neces¬ 
sarily  the  maior  ob|ective.  especially  if  this  requires  adding 
switches  and/or  communication  links,  which  consume 
silicon  real  estate  In  the  new  technology,  processors  will 
be  the-  expendable  components,  as  gates  were  in  SSI  or 
small  logic  networks  in  LSI 

This  mav  |ustify  different  fault-tolerance  schemes  which 
do  not  attempt  to  achieve  100-percent  utilization  of  the 
fault -free  processors  when  the  array  is  restructured  to  avoid 
the'  use  of  faulty  ones  [  1 8]  Such  schemes,  which  give'  up  the 
use  of  some  fault-tree  PEs  upon  restructuring,  can  be  attrac¬ 
tive  for  operational  faults  (which  are  few  in  number)  Here. 
the1  lack  of  additional  hardware  (switches  or  links)  allows  a 
larger  number  of  PEs  to  fit  into  the  same  chip  area,  thereby 
offsetting  the  penalty  of  giving  up  the  use  of  fault-free  PEs 
when  restructuring 

The  reported  research  in  this  area  of  fault-tolerant  archi- 
tectures.  although  a  significant  beginning,  is  limited  in  the' 
following  aspects: 

a  I  Most  of  the  proposed  architectures  have  been  devel¬ 
oped  on  an  ad  hoc  basis  No  well-established  criterion  or 
framework  yet  exists  for  the  formulation  of  these  architec¬ 
tures 

b)  As  indicated  above,  redundancy  can  be  used  for  both 
yield  enhancement  and  reliability  improvement  Recently, 
development  of  models  to  evaluate  how  can  a  given  re¬ 
dundancy  be  shared  to  achieve  the  best  combined  improve¬ 
ment  of  yield  and  performance  has  begun  (21)  but  more 
extensive  work  is  still  needed  Such  models  could  also  be 
used  to  compare  and  evaluate  different  architectures 

c  )  The  testabilitv  and  reconfigurability  issues  have  seen 
very  limited  treatment.  Algorithms  for  testing,  diagnosis, 
anci  reconfiguration  need  to  be  developed 


(c ) 

fig.  1.  Mesh  connected  arrays 
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III  A  Taxcinomv  for  Mut  iiproc  f ssc  >r  Arc  nine  itiRts 

Broadly  there  are  two  types  of  interconnection  arc  hitec  - 
lures  that  are  of  interest  to  VLSI  processor  array  implemen¬ 
tation  The'  first  type  is  the  nearest  neighbor  interconnec¬ 
tion  which  includes  various  mesh  interconnections,  as 
illustrated  in  Fig  I  The  second  tvpe  we  refer  to  here  as 
algebraic  graph  networks  which  includes  networks  such  as 
Erinary  n-c  uhe.  cube-connected  cycles  shuffle-exchange 
graph  shift  and-replac  e  graph  networks  and  group  graph 
networks  L samples  of  the'  latter  arc'  illustrated  in  f tg  like 
the  mesLi  c  onnec  tion  networks  these-  admit  c-ffic  ic-nt  exi  c  u 
turn  of  certain  algorithms  Also  algebraic  structure  of  some 
rrl  these  networks  can  fie  exploited  so  as  to  realize  asvrtip- 
tutu  alls  optimum  V  l  SI  layouts 

In  order  to  represent  uniformly  different  types  of  sue  h 
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architectures,  using  different  types  of  processing  nodes 
(processors  with  internal  switches  and  processors  with  ex¬ 
ternal  switches)  and  different  types  of  switches  (switches 
used  for  routing  and  switches  used  for  fault  detection  and 
reconfiguration),  we  present  the  following  taxonomy  Gen¬ 
erally,  there  are  two  types  of  system  nodes  nodes  capable 
of  only  computation  and  nodes  capable  of  both  computing 
and  switching  for  routing  In  addition,  there  are  two  type's 
of  switches,  the  conventional  switches,  capable  of  only 
establishing  connections,  and  fault-detecting  sv\ itches, 
those  that  perform  the  function  of  both  fault  detection  and 
reconfiguration  Different  types  of  architectures  are  delin¬ 
eated  in  fig  3  The  advantage,  generally,  in  using  external 
switches  is  that  the  computational  space  can  be  distinct 
from  the  communication  space  which,  therefore,  provides 
greater  flexibility  for  emulation  of  a  variety  of  communica¬ 
tion  geometries  The  disadvantage  of  external  switches, 
though,  is  that  they  require  additional  hardware  support 
and  occupy  extra  VLSI  area 

Different  types  of  architectures  are  illustrated  in  fig  3 
First,  Fig  3(a)  illustrates  an  architecture  where  the  PEs 
perform  internally  all  the  switching  necessary  to  establish 
c onnec tions  Fig  3(b)  represents  an  architecture  where  all 
the  connec  tions  are  established  by  using  external  switches 
Such  differences  are  best  illustrated  by  using  the  follow¬ 
ing  5-tuple  representation  of  networks  Let  N  =  (P.  S, 
£,,.  £,,  f(t  ,)  denote  the  network,  where  P  represents  the 
sc't  of  PEs,  S  denotes  the  set  of  switches,  f  denotes  the'  set 
of  direct  processor- processor  links,  £.  denotes  the'  set  of 
direct  switch-switch  links,  and  £  denotes  the  set  of 
processor- switch  links  Different  architectures  can  be  con¬ 
veniently  categorized  into  the  following  four  types,  as 
shown  below,  where  <f>  represents  the  null  set 
Ivpe  I 

(P.S =  <*>,£„,£.=  £,,,  =  *>. 

This  denotes  the  type  of  architecture  shown  in  Fig  3(a) 
Here,  the  array  contains  only  processing  nodes  with  switc  hes 
built  in  as  an  integral  part  of  the  processor  The'  mesh 
connections  considered  in  [18]  is  an  example  of  such  an 
architecture 
Type  1 

<P,S  £„  =  <(..£.,£„  .) 

This  denotes  the  type  of  architecture  shown  in  Fig  3(b) 
where  all  of  the  configuration  and  communic  ation  func¬ 
tions  are  performed  by  switches  that  are  external  to  the 
processor  The  CHIP  architecture  proposed  by  Snvder  [4l[  is 
an  example  of  this  type 
Type  3 

<P.S,  £„.£.=  *.£„.> 

Fig  3(c)  delineates  such  an  architecture  Here,  m  addition 
to  the  external  switches,  each  processor  has  an  internal 
switch  which  sets  up  the  connections  between  processors 
The  external  switches  are  used  to  provide  the-  function  of 
fault  detection  through  disagreement  detection  and  subse¬ 
quent  switching  out  of  the  faulty  processor  thus  discon¬ 
necting  it  from  the  network 
Tvpe  4 
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This  donates  a  tv pe  of  architectui  where  all  of  the 
different  types  ot  links  are  used  An  example  of  such  an 
architecture  is  illustrated  in  fig  3(d)  Here  a  linear  arrav  of 
Pts  is  provided  with  external  switch  connections  which  can 
he  configured  in  four  wavs  as  shown  in  Fig  4(a)  The 
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Fig  4  (a)  Different  switch  configurations  (b)  linear  arrav 

and  binary  tree  configurations  < c )  Bypassing  the'  faults  PE 


switches  in  such  an  architecture  have  a  dual  purpose  First, 
thev  can  be  used  to  provide  multiple  logical  configurations 
such  as  binary  tree  in  addition  to  the  linear  array,  thus  an 
a(  plication  that  requires  both  linear  arrav  and  binary  tree 
can  use  this  architecture,  as  shown  in  Fig  4(b)  Secondly, 
the  switches  can  be  used  to  bypass  the  faulty  elements,  as 
shown  in  Fig  4(c). 

As  we  can  see,  these  different  categorizations  encompass 
all  of  the  different  possible  architectures  that  can  be  con¬ 
ceived  Therefore,  the  above  taxonomy  provides  a  conve¬ 
nient  framework  for  both  the  analysis  of  different  architec¬ 
ture's  and  the  conceptualization  of  new  architectures 

There  are  two  basic  ways  one  can  introduce  fault  toler¬ 
ance  into  these  arrays,  the  first  approach  would  be  to 
provide  redundancy  at  each  node  so  that  the  node  can  be 
reconfigured  internally  in  the  event  of  a  fault  For  example, 
consider  a  9-node  mesh  connection  shown  in  Fig  5  If  we 
assume  that  the  interconnects  are  highly  reliable,  one  wav 
to  design  this  array  so  that  it  will  be  fault-tolerant  is  to  use 
two  self-checking  processors  at  each  node,  as  shown  in 
Fig  6  The  function  of  the  external  switch  is  to  determine, 
in  th  e  event  of  a  fault,  which  one  of  the  two  checkers  is 
indicating  errors  and  then  to  switch  out  the  appropriate 
module 

However,  if  the  interconnects  cannot  be  assumed  to  be 
reliable,  one  has  then  to  provide  redundancy  by  designing 
an  arrav  larger  than  the  maximum  size  required  for  the 
applications  For  example,  consider  the  4x4  array  shown 
in  Fig  7  which  is  designed  to  support  various  applications 
including  the  binary  tree  configuration  shown  in  Fig  8(a) 
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Fig.  5.  A  9-nodp  mesh  connection 
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F'g  7.  A  4  x  4  mesh  connec  tion 


The  mapping  of  the  binary  tree  onto  the  arrav  is  depicted  in 
Fig  8(b)  In  this  figure,  the  mapped  nodes  of  the  binary  tree 
are  shown,  along  with  the  inactive  components,  which  are 
shown  by  dashed  lines  Consider  now  that  the  active  node 
b  becomes  faulty  It  can  be  easily  seen  that  the  network  can 
no  longer  admit  the  binary  tree  configuration,  shown  in 
Fig  8(a)  However,  should  it  be  possible  to  execute  the 
same  application  on  a  reduced  binary  tree  (perhaps  with  a 
degraded  performance)  such  as  the  one  shown  in  Fig  9.  the 
application  can  still  be  supported  by  the  faulty  arrav,  as 
demonstrated  below 

There  are  two  different  ways  this  can  be  achieved  First, 
the  original  4x4  array  can  be  restructured  into  a  smaller 
3x3  array,  as  shown  in  Fig  10  This  would  require  giving 
up  the  use  of  some  processing  nodes  by  turning  them  into 
connecting  elements  (CEs)  [18)  Then,  any  application  that 
can  be  executed  on  a  3  x  3  array  can  be  executed  on  this 
now  (logical)  3x3  array  The  second  approach  would  be  to 
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Fig  9.  Redue  ed  binary  tree 


map  directly  the  application  configuration  onto  the  faulty 
physical  array  However,  the  latter  approach  can  he  compu¬ 
tationally  complex  (9j  Thus  depending  on  whether  or  not 
such  reduction  is  possible,  the  network  may  or  may  not  be 
fault-tolerant,  with  respect  to  this  application 
Several  important  concepts  emerge  from  the  above  dis¬ 
cussion  First,  a  node  or  link  can  assume  several  distinct 
states  The  following  shows  various  possible  states  of  the' 


Fig  10  Rodui  ed  j  X  j  jrrj\ 
node: 


Her<>,  the  processing  state  of  the  node  refers  to  that  -1,111' 
in  which  the  node  is  assigned  to  perform  some  useful 
c  omputational  task 

On  the  other  hand,  a  node  in  the  transmission  state  is 
assigned  to  perform  only  switching,  so  as  to  establish  a 
path  Thus  a  node  in  this  state  does  not  perform  any 
c  omputations,  except  those  which  may  be-  required  tor 
routing,  etc.  For  a  link  though,  this  distinction  does  not 
apply  Accordingly,  there  are  feyver  states  for  a  link,  as 
shown  below 


The  various  possible  state  transitions  an1  shown  by  the' 
following  directed  graph  Here  F  P.  T  4.  and  /  denote  the 
faulty,  processing,  transmission  active.  and  inactive  states 
respectively  The  arc  labels,  >  and  1,1.  represent  the'  transi 
tions  caused  by  fault,  and  change'  of  application,  respec¬ 
tively 


Secondly,  the  various  reconfiguration  processes  can  be 
conceptualized  through  an  abstraction  of  lasers,  formulated 
below 

Let  the  ph\sudl  /aver  represent  the  topology  which  de¬ 
scribes  the  interconnection  structure',  along  with  the  status 
of  the  nodes  and  links  in  the  physical  array  A  node,' link  in 
the  physical  layer  can  bo  either  in  the  fault-free  or  faulty 
state 

Let  an  jpplnjtion  lavrr  represent  that  topology  yyhich  is 
requited  to  support  a  giyen  application  Thus  in  this  layer, 
all  of  the-  nodes  are  processing  nodes:  the  links,  active  links 

let  the  logit  jl  l.ner  represent  the  topology  yvhich  real¬ 
izes  a  giyen  application  layer  on  a  given  physical  layer  Thus 
a  node  in  this  laser  is  either  in  the'  processing  state  or  in  the 
transmission  state  All  of  t Lie  links  in  the  logical  layer  are  in 
the  a<  tiso  state 

for  a  giyen  configuration,  the  above  layers  are  related 
topologically,  as  shown  in  Fig  I  I  The  nodes  in  the  applic  a- 


-  Physical  layer 


Logical  layer 


Application  layer 
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tion  layer  are  a  subset  of  the'  nodes  in  the  corresponding 
logical  layer  and  these  are.  in  turn,  a  subset  of  the  nodr's  in 
the  physic  al  laser 

Tlie  following  defines  a  set  of  fundamental  problems  of 
prac  tic  al  importance 

Prohlt'm  I  Given  an  application  layer  (a  set  of  applica¬ 
tion  layers)  and  the  physical  array  that  admits  these  applica¬ 
tion^)  what  is  the  minimum  size  (number  of  nodes,  silicon 
area)  of  the'  physical  laser  that  can  admit  the  applic ation(s) 
when  t  or  fewer  components  faib 

Prohlt'm  J.  Given  the  geometrical  structure(s)  of  an 
application  layer  (set  of  application  layers),  how  can  a 
physical  array  be  designed  so  that  it  can  proside  "efficient" 
fault -tolerant  realization  of  the  applic ation(s)?  The  term 
efficient  mas  be  ciefined  in  terms  of  factors  such  as  size  of 
physical  array,  length  of  c  ommunic  ation  delay  between 
ad|acent  application  nodes,  ease  of  testing  arid  diagnosis 
rec  onfigurabilitv.  etc 

The  above  problems  need  to  be  studied  in  the  context  of 
more  general  and  flexible  use  of  redundancy  for  example 
judicious  use  of  node-level  redundancy  mas  offset  the 
need  for  massive  reciundanc  .  at  the  system  level  Also 
broader  use  of  switches  as  implied  by  Type  i  and  Type  -1 
arc  hitec  tures  mas  yield  new  system  arc  hitec  tures  arc  hitei 
lures  that  provide  more  efficient  utilization  of  redundancy 

The  above  discussion  is  also  applicable  to  the  sec  on.) 


type  of  networks  the'  algetiraic  networks  For  example 
consider  the  xhiftand-rrplac  e  graph  networks  proposed 
recently  m  [3d]  as  a  candidate  for  '  LSI  processor  networks 
Such  an  8  node  network  is  shown  in  Fig  l.’(a)  Ihis  net 
work  is  capable  of  emulating  various  useful  logical  strue 
fun's  such  as  the  linear  arras,  binary  tree'  shuttle  and  the 
shuffle-exchange  communication  strue  tures  as  shown  m 
Fig  1 2(b)  More  importantly,  this  algebraic  network  can 
emulate'  structures  such  as  the  linear  arras  and  binary  tree, 
in  spite  of  a  fault  For  example,  consider  the  link  Conner  ting 
nodes  1  and  2  becoming  faulty  In  this  case'  the'  networks 
can  still  be  restructured  Lioth  as  a  linear  arras  and  as  a 
binary  tree',  as  shown  in  Fig  13  Similarly  the  network  is 
also  capable  of  emulating  these  structures  in  spite-  of  any 
single-node  failures 

It  may  also  be  noted  that  networks  such  as  the  binary 
n-cube  and  the  cube-connected  cycle's  provide  some  inter  - 
esting  fault-tolerant  reconfiguration  capabilities  For  exam¬ 
ple.  consider  a  4-cube  of  16  nodes,  shown  in  Fig  14(a)  In 
the  event  of  a  fault,  one  can  degrade  this  to  a  3-cubi'  of  8 
nodes,  as  illustrated  in  Fig  14(a)  However,  this  would 
require  giving  up  the  use  of  seven  good  nodes  Alterna¬ 
tively,  one  can  partition  the  4-cube  into  4  subnetworks  of 
2-cubes  Assuming  that  the  problem  can  Fir'  divided  into 
subproblems  that  can  be  executed  on  2-cuties.  one  can  use 
3  of  these,  as  shown  in  Fig  14(b)  This  would  nec exxitate 
giving  up  the  use  of  only  3  good  nodes  It  is  obvious  that 
the  fault  tolerance  of  algebraic  networks  can  be  studied  m 
the  context  of  VLSI  processor  arrays 


IV  Tfsnisic  and  Riconfiguration  Sirahciis 

Central  to  the  success  of  any  fault-tolerance  scheme-  is 
the  formulation  of  effective  testing  and  reconfiguration 
strategies  Basically,  there  are  two  different  approaches  to 
diagnosis  and  recovery  centralized  and  distriliutr-d  In  a 
centralized  procedure,  one  may  assume  an  external  unit 
which  is  responsible  for  initiating  testing  and  reconfigura¬ 
tion.  In  a  distributed  procedure,  the  PEs  themselves  are 
responsible  for  performing  periodic  testing  and  rec  onfigura- 
tion 

The  advantage  of  a  centralized  scheme  is  that  no  ad¬ 
ditional  hardware  and  software  support  has  to  be  provided 
within  each  PE  to  allow  testing  and  reconfiguration  On  the 
other  hand,  useful  computation  for  the  entire  arras  has  to 
bo  interrupted  so  that  testing  can  be  performed  Addition 
ally,  the  complexity  of  the  circuit  anci  the  limited  .im1", 
from  the  external  unit  mas  not  allow  a  centralized  pmu 
dure  to  be  used  The  advantage  of  distributed  texting  on 
the  oilier  hand,  is  that  since  eac  h  processor  can  peitoim 
texting  m  an  asynchronous  mode  the  texting  can  tie  inter 
leaved  with  computation,  thus  not  net  "xsarilv  requiting  a 
complete  interruption  of  all  useful  computation  Moreover 
(tie  distributed  testing  has  the  potential  tor  t  •  it.  r  ’acilt 
i  overage  Fiei  auxe  of  the  proximity  ot  the  testing  urn!  and 
the  unit  under  text 

From  ttie  utilise  dixe  uxxion  it  is  apparent  that  a  lixtub 
cited  procedure  muxt  strive  to  make  the  testing  and  n-mi 
figuration  task  loc  a  I  to  eac  Fi  node  I  tux  was  the  texting  and 
reconfiguration  can  tie  made  transparent  to  most  ot  It,. 
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network  Howesrr  performing  those  tasks  lor  alls  requires 
extra  hardware  and  software  support  at  ear  h  node  and  a 
distributed  procedure  must  trs  to  minimize  it  On  tire  other 
hand  a  centralized  procedure  must  attempt  to  minimize 
the  number  ot  tests  that  will  be  required  when  no  faults  are 
present  Interruption  of  useful  r  (imputation  will  tie  this  was 
minimized 

In  the  following  we  present  an  example  tor  a  distributed 
testing  procedure  m  which  exerx  P(  tests  all  its  immediate 
neighbors  In  this  was  faults  Pis  and  faults  c  nimi'i  linns 
between  p(  s  ari  deter  ted  bs  the  adtac  ent  Pis  the  pme  e 
duri  first  partitions  all  the  Pt  s  into  m  disjoint  testing 


groups  7  7  7,  After  this  partitioning,  thrve  arr'  rn 

ptiases  ot  testing,  sshr're  at  phase'  i  (0  ^  r  ^  m  I)  the 
members  of  7  test  all  their  neighbors 

The  partition  is  sue  h  that  I  losers  Pf  is  surrounded  bs  Pt  s 
ot  other  grou"s  and  .’)  no  PE  has  tsso  neighbors  belonging 
to  tire  same  group  These  tsso  properties  guarantee’  that  tor 
e\erx  i  no  tsxo  members  of  7  sxill  test  ear  ft  other  or  tis 
simultaneousls  to  test  a  third  Pt  It  r  an  easilx  tie  stiossn  tfiat 
tise  (si  sen  |  grrnips  are  both  net  essars  and  sufficient  tor  a 
partition  sxith  t tie  abuse  properties  in  the  r  ase  ot  a  square 
arras  |IH]  (hexagonal  arras  { id])  The  testing  group  numbers 
assigned  to  ear  fi  Pt  m  a  square  arras  and  an  hexagonal  arras 
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Several  problems  related  to  testing  and  reconfiguration  ot 
these  arravs  have  been  dest  nbed  Both  the  distributed  and 
<  entrali/ed  modes  ot  testing  have  been  considered 

The  last  part  ot  the  paper  is  devoted  to  the  presentation 
of  anahtual  models  tor  the  •'valuation  ot  reliabilitv  arid 
vield  improvement  through  redundant  \  T  tie  available  re 
dundan<  v  on  the  i  hip  or  water  is  pnmanlv  limited  hv  tU» 
si/e  ot  tt)*'  i  tup  or  water  hence  it  is  imperative  to  find  1 
method  hv  which  one  i  an  optimalh  share  the  available 
redundjm  v  lietwi'en  \ielit  enhancement  and  pertormjni  » 
improvement  The  models  discussed  *  an  tie  used  to  studv 
the  ettei  t  ot  stuping  av aitabie  redundant  v  betw»-en  ttiese 
two  somewhat  <  ompeting  rerjuin  rn*  nts 
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